Skip to content

intermediate representation for directory trees / filesystems with fine control over metadata

License

Notifications You must be signed in to change notification settings

malt3/abstractfs

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

abstractfs

Go Report Card GoDoc

intermediate representation for directory trees / filesystems with fine control over metadata

Quick start

go install github.com/malt3/abstractfs@latest
abstractfs json --source-type tar --source /path/to/archive.tar | yq -P
abstractfs json --source-type dir --source /path/to/directory | yq -P
abstractfs convert --source-type dir --source /path/to/directory --sink-type tar --sink /path/to/archive.tar

Architecture

flowchart LR
    transform["transformations (strip metadata, ...)"]
    source
    sink
    json
    ir --> transform --> ir
    subgraph ir[intermediate representation]
        direction TB
        root["/"]
        etc
        usr
        passwd
        resolv["resolv.conf"]
        root --- etc
        etc --- passwd
        etc --- resolv
        root --- usr
    end
    source -->|nodes| ir -->|nodes| sink
    ir --> json
Loading

Design goals

  • Simple
  • Arbitrary input and output formats
    • directory of a mounted filesystem (dir)
    • archive formats (tar, cpio, zip)
    • package formats (rpm, deb)
    • container image formats (oci image, oci layer)
    • filesystems (squashfs, fat, ext4)
    • go fs.FS (embed.FS)
    • in-memory sources and sinks
    • user-extensible, programmable via an interface
  • Efficient in-memory representation
  • Clean interfaces for sources and sinks
  • Human readable metadata representation
  • File contents stored in abstract CAS and can come from anywhere (even other computers)
    • represent hash using Subresource Integrity Format
    • File tree operates on nodes containing only file metadata and SRI. Contents handled out-of-band.
  • Composable (support overlaying / nesting / merging operations)
  • Support reproducible / repeatable operations / transformations as event stream
    • add / remove a file
    • replace contents or metadata
    • serializable event format
  • Uses canonical representation by default
  • Explicitly support the absence / presence of metadata
  • Should allow for metadata normalization operations
    • Strip attributes, normalize timestamps...
    • Overlay metadata from a different source
  • Designed for file system forensics / diffing / reproducible builds

Supported sources and sinks

Source Sink xattr CAS Source
dir 🔜
go fs.FS 🔜
tar
cpio 🔜 🔜 🤷 🤷
zip 🔜 🔜 🤷 🤷
rpm 🔜 🔜 🤷 🤷
deb 🔜 🔜 🤷 🤷
oci 🔜 🔜 🤷 🤷
squashfs 🔜 🔜 🤷 🤷
fat 🔜 🔜 🤷 🤷

Content addressable storage (CAS) backends

  • in-memory
  • dir
  • S3 / object storage

🚧 JSON Format

The current format is not stabilized and still work in progress.

Core

The core is implemented in a separate module and has no dependencies.

About

intermediate representation for directory trees / filesystems with fine control over metadata

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages