LayerFS: 157 snapshots in 100 MB, read from the code and rebuilt as a toy

mdjsonmcpcopy↗ai

2026-10-27 · 19 min · agents · developer-tools · storage · performance · benchmarks

Why read this

Solid· top 85%

Reads the chunker, tree limits and merge code and measures a toy store over 157 real commits. Cannot confirm the headline 100 MB figure, and says so.

• Original analysis
• Runs on a laptop CPU
• Concrete numbers to act on

·Developer tools & infra·MIT·Practitioner tool

How this was scored

Is it new?

1 of 3: An incremental tweak

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

2 of 3: Open code or weights with real limits

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

1 of 3: Relevant for months

Does it affect many?

1 of 3: A specialist community

Only here?

2 of 3: A teardown or measurement few others did

Score 58 of 100, ranked 415 of 621 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

LayerFS is a filesystem for agents. Each agent gets a complete, writable copy of a project, the copy costs almost nothing to make, and what the agent changes is saved as a commit on a branch in a store every agent shares. The README sells ten ideas. One of them is a number: 157 snapshots of a real project in 100 MB, 19.9x smaller than Btrfs with zstd. The author's announcement post says the same thing in French. I could not open the post (the page returned HTTP 402), so I work from the README and the research notes in the repository, which carry the numbers.

This article does three things. It reads the code to say what the store actually is. It rebuilds the idea as a small Python toy and measures it on 157 commits of a repository I have, with real byte counts. And it says plainly whether the 157-in-100-MB claim could be checked from here. It could not, and the reasons are below.

What problem it is aimed at

An agent that edits code needs a working directory, and parallel agents need one each. A git worktree per agent drops node_modules and build output, and a copy is a copy. LayerFS mounts one shared committed tree as a Workspace: writable, with its own overlay for changes. When the command finishes the changes are committed and the mount goes away. If every tool call becomes a commit, history grows by one snapshot per call, and the question is what a snapshot costs.

The model, as the code and docs state it

This section is read, from docs/general/concepts.md, the core/ crates and the engine/ crates.

• Store. One SQLite file holds every branch, commit and content object. Daemons and the SDK open it directly; there is no database server. The store runs in a "Disposable" profile only, with no crash or power-loss durability. The README says this itself.
• Layer, branch, commit. A layer is an immutable published snapshot. A branch is a named writable line based on a layer. A commit is an immutable snapshot of a branch. Publishing a branch to main adds a layer. Forking a branch and publishing a layer reuse existing roots and copy no payload.
• Workspace. An ephemeral view of one branch head. layerfs-overlay describes itself as "one Workspace's disposable SQLite database". Commit captures the final state and publishes with a compare-and-swap on the branch head. End never commits implicitly, and a clean End refuses uncommitted state.
• Objects. Files and trees are immutable canonical objects. An object's ID is a BLAKE3 digest over a frozen domain separator (layerfs/object/v2) followed by the canonical bytes. Objects are append-only; inserts deduplicate on the ID; normal operations never delete.

The chunker

A file is cut into pieces by content, so that an edit disturbs only the pieces around it. file/cdc/gear.rs is a frozen two-byte rolling GEAR chunker, and its parameters are constants:

The first 8 KiB of every chunk are skipped without scanning. A stricter mask applies below 16 KiB, a looser one above, and a boundary is forced at 32 KiB. Two consequences, reasoned from the code: a file under 8 KiB is always one chunk, so for small files this is whole-file addressing; and a boundary depends only on the bytes since the previous one, so the chunker resynchronizes after an edit.

The profile is frozen on purpose: changing any constant or table entry changes every chunk identity, so profile_id() hashes all of them.

Trees, pages and encodings

Directories and file-to-chunk maps are also objects. From filesystem/limits.rs (read): a page is at most 8,192 bytes; an inode leaf holds 50 to 100 rows; a branch holds 64 to 127 children. That is a wide, shallow, B-tree-shaped structure, which matters for the next section: changing one file rewrites one leaf and the path of branches above it, not the whole directory.

Objects are then packed. layerfs-storage has four pack lanes (ordinary, chunk, whole-file, singleton). Payload frames are compressed with zstd level 3. Groups of value records are compressed with level 1 and capped at 65,536 bytes. A stored record can be a FULL representation, a PREFIX (a payload encoded against a base payload) or a DELTA against another object. The project's own report counts 28,106 FULL and 58,306 PREFIX records and 4,583 inode-leaf DELTAs in its September research run (reported).

Merge

Merging is three-way over immutable roots (merge/, read). Two commits that change the same file are refused with Conflict unless a rule says otherwise. The conflict kinds in ConflictKind are Content, AddAdd, ModifyDelete, DeleteModify, Type, Binary, Markers and Mode. Text files get a line-level three-way merge with markers. Anything binary keeps "ours" unmarked and reports a conflict. A /.lww file holds gitignore-style patterns; a path both sides changed that matches takes the side with the newer modification time, ours on a tie. That is the escape hatch for lockfiles and build outputs, where two agents will always collide and no human wants to resolve it.

The storage side is where the claim lives.

The claim, and what it would take to check it

The README's results table says (reported): "A real project's history, 157 snapshots (4.94 GB of trees), kept as mountable Commits. 100.1 MB vs 1,994.6 MB, 19.9x smaller than Btrfs + zstd; 156/156 read back."

The arithmetic holds. 1,994.6 / 100.1 = 19.9, and 4,936,693,030 logical bytes over 100.1 MB is 49.3 to 1. The note beside the table says one sample per case, an arm64 Mac running Docker Desktop, 8 CPUs, 8.3 GB.

A second document, docs/research/layerfs-vs-btrfs-157/REPORT.md, covers the same 157 checkpoints in September. Its LayerFS figure is 184.6 MB, from an earlier research candidate, and its ratio against the same Btrfs baseline is 10.81x. So the headline moved by a factor of 1.8 between that candidate and the 1.0.0 release (reported). The README links the report.

The workload is 157 Git tree states of DeepSeek Harness, picked at ordinals 100, 200, ... of 15,632 commits, with .git, dependencies and build output excluded. 904,143 path occurrences, 4.94 GB of logical content, 891.9 MB of unique Git blob bytes. About 65% of unique file contents are under 8 KiB. Changed files are rewritten in full on every step, 181,002 writes totalling 1.72 GB, with no hardlinks, reflinks or byte-range patches helping either side.

Could I check it? No, for three separate reasons.

1. I did not run LayerFS. I read the Rust and did not build the workspace. The real thing mounts through FUSE in a Linux daemon, and the project's own runs used Docker Desktop on a Mac.
2. I do not have the inputs. The frozen manifest points at 15,632 commits of another repository, and its unique blobs alone are 892 MB. My scratch budget for this article was 600 MB.
3. The comparator is not mine either. The Btrfs numbers came from loop devices in a privileged container, with compress=zstd:3 and metadata DUP.

What I can say from the report itself, reported: the authors flag that LayerFS's figure is a full-store allocation including indexes, that Btrfs's includes DUP metadata copies, and that the two "are not identical whole-device accounting" and do not establish equal durability. Their ranking of Btrfs is also workload-bound: on a separate 512 MiB file, changing 4 KiB through a Btrfs snapshot cost about 36 KiB. They say this does not transfer to the small-file history.

So the claim is a specific, carefully caveated measurement of one workload. What I could do is ask whether the idea behind it produces numbers of that size on a different workload.

Experiment: a toy of the layer-and-dedup idea

The setup, all measured:

• Workload: 157 commits sampled evenly, oldest to newest, from the git history of this site's repository (728 commits over 19 days), with public/ excluded because it is 1.6 GB of images and video that are not what the experiment is about. 569,351 file occurrences, 7,795,062,839 logical bytes. Different workload from LayerFS's: a Next.js site with MDX, TypeScript, JSON and vendored JavaScript, not DeepSeek Harness.
• Only 4,525 distinct files (by git blob) appear across the 157 snapshots, 132,433,378 bytes in total.
• The toy chunks each distinct file with a Python port of the frozen GEAR chunker, reading the 256-entry table and the four masks out of gear.rs. Files of 8 KiB or less are one chunk. Chunk IDs are SHA-256 (the real store uses BLAKE3). Each unique chunk is compressed on its own with zlib level 6. zlib stands in for zstd everywhere (my Python has no zstd), so ratios between rows are meaningful and absolute sizes will not match LayerFS.
• The toy stores each directory as an object listing its entries and each file's chunk IDs, and addresses that object by hash, so an unchanged subtree is stored once. Each snapshot adds a 100-byte commit record. Every object carries 36 bytes of index cost. That is a git-like Merkle tree, not LayerFS's paged one.
• Comparators: plain copies (sum of file sizes); a hardlink farm (each distinct file once, uncompressed, directory entries not counted, which flatters it); one tar.gz per snapshot; and one tar of the distinct files through xz.

The port was not cross-checked against the Rust chunker's output. I read the scan code and ported it, but I have no golden vector. A sanity check I did run: on 3,002 chunks from large files the mean chunk was 19.8 KB, p10 12.6 KB, p50 18.7 KB, p90 28.9 KB, with 5.4% cut at the 32 KiB maximum. That is the right shape for an 8/16/32 profile, and it is the only validation I have.

All sizes at 157 snapshots, measured unless marked:

The tar.gz row is a scale: I summed per-file zlib sizes over every snapshot (3,175,421,219 B) and multiplied by 0.895, the ratio I measured on the last snapshot between a real tar.gz (19,623,863 B) and the same files compressed one by one (21,923,627 B).

Three readings.

The hardlink farm gets most of the way. On this workload 59x comes from storing identical files once, because the snapshots are mostly the same files. A headline against plain copies would be uninteresting, which is why the project compares against Btrfs.

Compression is the next factor of three, and chunking is the smallest. Compressing the distinct files takes 132 MB to 38.6 MB (3.4x). Chunking cuts the raw bytes you must hold by 44% (73,793,323 against 132,433,378, because a changed large file shares most of its pieces with its previous version), but after compression the payload falls only from 38.6 to 27.5 MB, 29%. Compression had already removed much of the redundancy chunking would have removed. For a text-heavy workload like this one, that is a reasonable result; for big binaries, where compression finds little, I would expect chunking to matter more. That last sentence is reasoned, not measured.

The archive beats the toy. One xz archive of the distinct files is 20.4 MB, half the toy's 41.6 MB. It is also a single file with no random access, no history structure, and no way to mount a snapshot. The toy pays for those properties, and the next experiment shows where.

Experiment: the edit that touches a big file

The README's idea ten is "an edit costs the size of the edit, not the size of the file." With a content-defined chunker, an edit rewrites the chunks around it. I took the two largest files in the sampled history (a 1.70 MB JavaScript example and the 0.99 MB minified p5.js this repository vendors), applied a 100-byte edit to the middle, chunked again, and counted chunks whose hash is new. Measured:

An overwrite costs one chunk, 1.3% of the larger file. The insert into the first file cost four, 4.4% of it, where the second file needed one. I do not know why the first file resynchronizes slowly; I looked at the boundaries only through the port, and a port bug is possible. What the numbers support is narrow: edits of this size to files of this size store tens of kilobytes, not megabytes, before compression. They do not say that the cost is "the size of the edit", which here is 100 bytes. The floor is one chunk, 8 to 32 KiB. For an edit loop of small changes that is roughly a 200x amplification of the edit and a 75x saving against the whole file.

Where the bytes go: structure, not payload

The ledger above hides the most interesting number in the toy. Split the 41.6 MB by what it is made of:

• chunk payload: 27,500,226 B (66%)
• flat directory objects plus commit records: 14,061,679 B (34%)

And the growth later in the history is worse for structure. Between snapshot 80 and snapshot 157 the toy grew by 145,942 B per snapshot, of which payload was 52,979 B and directory objects 92,863 B. Directory structure was 63% of the marginal cost of a snapshot. The first snapshot itself, 18.5 MB, is almost all payload (18.27 MB).

The reason is simple. Over the 157 snapshots the repository has 569,351 file occurrences, roughly 3,600 files in each. A snapshot changes a handful of them. But in my toy a directory is one object, so changing one file in a directory with 300 entries writes the whole 300-entry listing again, with every file's chunk IDs. The big directories are the ones that change most, so it happens often.

LayerFS does not do this (read): leaves hold 50 to 100 rows and branches hold 64 to 127 children, and a leaf can be stored as a delta against its predecessor. So I changed the toy. In the second variant a directory's sorted entries are split into pages with boundaries chosen by a hash of the entry name, about 64 entries a page, and the directory object lists page IDs. A one-file change rewrites one page plus the small directory object.

Measured: paged directories cost 4,577,755 B against 14,045,979 B flat. The total at 157 snapshots drops from 41.56 MB to 32.09 MB, and the marginal cost per snapshot after snapshot 80 from 145,942 B to 80,684 B. Structure falls from 34% to 14% of the store.

Now put LayerFS's own breakdown next to it. From the September report (reported), the 184,598,528-byte store is made of 98,256,041 B of payload groups (53%), 56,719,413 B of structure groups (31%), 378,096 B of pack headers, 26,357,298 B of database outside the packed blobs, and 2,887,680 B unattributed. The structure share, 31%, is the same size as my flat toy's 34%. It is a different workload and a different representation, so I draw no tight conclusion. The reading I will defend is this: in a history of small files with many snapshots, a store's size is decided at least as much by how it encodes the tree as by how it deduplicates the files. The report says as much about Btrfs: its metadata region was 1,293.8 MB of the 1,994.6 MB baseline, and a dedup tool that shrank data by 53% grew metadata by 414 MB and made the total worse.

If the 1.0.0 release got from 184.6 MB to 100.1 MB, I did not read how, and I will not guess at the mechanism.

What this does and does not say about 157 in 100 MB

My toy reached 41.6 MB flat and 32.1 MB paged on a different repository. That is neither a confirmation nor a contradiction. The ratio to logical size is 188x to 243x for me against 49x for LayerFS, which would be an alarming difference if the workloads matched. They do not, and the comparison I find fairer is the shape of the results rather than the ratio:

• My snapshots are larger on average (7.795 GB / 157 = 49.7 MB against LayerFS's 4.94 GB / 157 = 31.4 MB), but my history is text-heavy and comes from an actively edited repository with very few large new files.
• Snapshot spacing differs. LayerFS picked every hundredth commit of 15,632; I sampled 157 of 728, about 4.6 commits apart. Closer snapshots change less between steps, which should make mine cheaper.
• I did not count the real store's index and database overhead. My 36 bytes per object is a guess. Their 18.7 MB of objects-index pages is 10% of their research store.

So the honest verdict is: the idea is sound and the size is plausible, the claim is unchecked. A content-addressed, chunked, tree-structured store with delta encoding should land well below a hardlink farm on this kind of history, and my toy did. Whether it lands at 100.1 MB on the 157 DeepSeek Harness states, I cannot say.

What it costs

From the project's own pages (reported), without my reproduction:

• No durability. The Store runs in its Disposable profile only. Use it for agent work and its history, not as the only copy of anything you cannot rebuild.
• Sequential writes are slow. In the fio suite the README lists 0.31x for sequential write, 0.81x for sequential read and 0.40x for random write against libfuse passthrough, while random read/write mixes ran 1.49 to 2.23x. A build that writes big artifacts through a mount will feel the first number.
• Not full POSIX. The FUSE view omits FIFOs, sockets, device nodes, set-id bits and separate atime/ctime. pjdfstest shows zero unexpected failures but keeps 76% of overlayfs's passes.
• Compact encodings cost reads. The research report says compact representations can require decoding and dependency traversal, and that its capacity comparison does not price that in.

And a cost I reasoned: by design every tool call becomes a commit, so the history is as long as the agent's activity, and the store only grows because objects are never deleted in normal operation. 80 KB to 146 KB per snapshot in my toy is small for 157 snapshots and is not small for 157,000. I did not find garbage collection in the paths I read.

What I'd take from it

If you run several agents on one codebase, three things in LayerFS are worth copying even if you never use the product.

1. Separate payload from structure when you measure storage. My flat toy and my paged toy have identical payload bytes. They differ by 30% in total size.
2. Check what your snapshot gap is before comparing. A ratio at every-fifth-commit and a ratio at every-hundredth are not the same experiment.
3. Compare to the archive. A single xz of the distinct files was 20.4 MB. If your store is within 2x of that and gives you mountable, branchable history, it is doing its job.

The code is MIT licensed and core has no engine dependencies, which made the chunker easy to read and port: one 567-line file, most of it the GEAR table.

Reproduce the toy

The script is public/articles/layerfs/toy_store.py in this site's repository. It needs Python 3, git, a repository to sample and the gear.rs file from a LayerFS checkout, and it writes result.json. It holds every distinct file in memory (132 MB for my run) and took a few minutes on a shared 4-core VM at low priority.

git clone --depth 1 https://github.com/Ephemeral-AI-Lab/layerfs
nice -n 19 python3 -I toy_store.py 157 /path/to/repo layerfs/core/crates/layerfs-content/src/file/cdc/gear.rs

The core of it is the chunker. This is the port, trimmed to the loop:

def cdc(b):                      # frozen profile: min 8192, target 16384, max 32768
    n, out, s = len(b), [], 0
    while s < n:
        if n - s <= MIN: out.append((s, n)); break
        pos, h = MIN, 0          # the first 8 KiB of a chunk are skipped unscanned
        while True:
            a = s + pos
            if pos >= MAX: cut = a; break
            if a + 1 >= n: cut = n; break
            small = pos < TGT    # stricter masks below the target size
            h = ((h << 2) + (GEAR[b[a]] << 1)) & M64
            if h & (SSMALL if small else SLARGE) == 0: cut = a; break
            h = (h + GEAR[b[a + 1]]) & M64
            if h & (SMALL if small else LARGE) == 0: cut = a + 1; break
            pos += 2
        out.append((s, cut)); s = cut
    return out

What I did not do

• I did not build or run LayerFS, so no number in this article comes from its code running.
• I did not check the 157-snapshot figure, the 19.9x ratio, the Btrfs numbers or any benchmark in the README.
• I did not read the announcement post, so I do not quote it.
• I did not cross-check my chunker port against the Rust one.
• I did not read the release notes on how the store went from 184.6 MB to 100.1 MB.
• I did not test merge, .lww or concurrent agents, only read their types and docs.
• I did not look at read latency, mount time, or the cost of a Workspace.

An independent check is cheap for someone with a Linux box and Docker. Run the project's dsh-dedup suite from layerfs-benchmark and compare its receipts to the table in the README. I would be glad to read that.