Filesystems: ZFS

In the previous article , we explored Btrfs—a copy-on-write filesystem built around a single kind of B-tree, where every file, extent, checksum and chunk mapping lives as a tagged item in some tree, and snapshots fall out of the reference-counted extent design. Btrfs took a lot of inspiration from an older system that pioneered most of these ideas: ZFS.

ZFS started life at Sun Microsystems in the mid-2000s and now lives on as OpenZFS, ported to Linux, FreeBSD, illumos, and macOS. From the outside it solves the same problems as Btrfs—pooled storage, copy-on-write, snapshots, checksums, integrated RAID—but the shape underneath is genuinely different. Where Btrfs leaned on one universal B-tree node format and a single key shape, ZFS leans on something else entirely: a 128-byte block pointer that fully describes the block it points to, and a strict three-layer architecture stacked on top of it.

In this article we’ll walk ZFS bottom-up, building each layer on top of the previous one: how a pool is laid out on disk, how the uberblock anchors everything, how block pointers form a Merkle tree (a tree where every node holds the checksum of its children, so tampering anywhere breaks the chain up to the root), how datasets and snapshots really work, and finally how a read and a write actually flow through the system end-to-end. Many of the ideas will feel familiar from the Btrfs article—but the way ZFS expresses them is its own.

A heads-up before we start: ZFS is huge, and to keep this article from turning into a book we’re going to oversimplify in places. Diagrams will collapse whole subtrees behind a ... node, some object types will get a passing mention rather than a full treatment, and a few corners (encryption, dedup internals, the full ZIO pipeline, vdev removal, raidz parity math, …) will be glossed over or skipped entirely. The goal is a faithful mental model of how the pieces fit together, not a complete reference—if you want the full picture, the OpenZFS source linked at the end is the real source of truth.

A Different Kind of Filesystem

Just like Btrfs, ZFS doesn’t live on a single partition—it manages a whole pool of devices (a zpool), and the filesystem you mount is just one of the things that pool can host. But what’s inside that pool is where ZFS gets interesting in its own way.

The thing that makes ZFS click, more than any single feature, is realizing it’s built as three stacked layers. We’ll dig into each one in detail later, but let me introduce all three up front—that way you have a mental shelf to put every new concept on as we go:

• SPA — the Storage Pool Allocator. This is the bottom of the stack, and the only layer that actually talks to disks. You hand it a pile of physical drives, and it arranges them into a tree of virtual devices (vdevs). Some vdevs are just a single disk; others are groups of disks bonded together for redundancy (mirrors and the RAIDZ family). The SPA also runs the I/O pipeline that pushes bytes on and off the platters—a staged async machine called ZIO that handles checksums, compression, encryption, and allocation as data flows through it.
• DMU — the Data Management Unit. The middle of the sandwich. The DMU’s job is to hide all that disk and vdev stuff behind a single, uniform idea: everything ZFS stores is an object, identified by a 64-bit number, and every object is typed, variable-sized, and copy-on-write. Files? Objects. Directories? Objects. The pool’s own bookkeeping? Also objects. This is the layer where copy-on-write actually happens—the SPA just allocates blocks; the DMU is the one that decides never to overwrite them.
• DSL — the Dataset and Snapshot Layer. The top of the stack, and the only layer you ever see as a user. It bundles objects into datasets—filesystems, volumes (zvols), and snapshots—and stitches them into the namespace you see on the command line (mypool, mypool/home, mypool/home@yesterday, and so on), complete with property inheritance from parent to child.

Everything you’ll ever do with ZFS travels through this stack, from the top down. Take a simple read(): the DSL turns the path you opened into a specific dataset (a filesystem); the DMU finds the object inside that dataset and walks its block pointers; the SPA turns those pointers into actual reads from real disks. Snapshots ride the same elevator: they’re a feature the DSL exposes, built on the DMU’s copy-on-write objects, which in turn rely on the SPA’s “allocate new blocks, commit at the end” rule. Keep this top-down layering in mind—it’s the spine everything else in the article hangs off.

Let’s start at the bottom, because the SPA’s shape ends up dictating what every layer above it can do.

The SPA: A Tree of Vdevs

The SPA is what turns a pile of disks into something the rest of ZFS can use. The first thing to know: a ZFS pool is not a flat list of disks. It’s a small tree of vdevs.

Picture it like this:

• At the leaves, physical disks—or occasionally files or partitions pretending to be disks.
• One level up, top-level vdevs: each one is either a single disk on its own, or a redundancy group built from several disks—a mirror (n-way replication), or a raidz1/raidz2/raidz3 (single, double, or triple parity).
• At the root, the pool itself, which stripes across all of its top-level vdevs.

So the pool is a vdev made of vdevs made (sometimes) of more vdevs that bottom out at disks. In real life it’s almost always exactly two levels deep—pool over top-level vdevs over disks—but the on-disk format is happy to describe a fully general tree, which is why oddities like “a mirror of raidz vdevs” are technically representable, even if you’d rarely see them in production.

Now here’s the rule that everyone who runs ZFS eventually internalizes: redundancy lives at the vdev level, never at the pool level. The pool stripes across its top-level vdevs purely for performance and capacity—that striping is not protecting your data. If you lose both disks in mirror-0 up above, or two disks in raidz1-2 (single parity), the entire pool is gone, even though the other top-level vdevs are perfectly fine. The pool only protects you against losing parts of a top-level vdev, never against losing the whole thing.

That covers the shape of the pool. The SPA still has two jobs left here—recognizing which physical disks make up the pool at boot, and deciding where on those disks new blocks go—plus a third one (recording the pool’s state) that’s important enough to deserve its own section right after.

The Vdev Label and Pool Discovery

Every disk in the pool carries four copies of a 256 KiB structure called the vdev label, two at the very start of the device and two at the very end.

The four-copy redundancy is deliberate: lose the head of the disk to a head crash and L2/L3 at the tail still anchor the pool; lose the tail and L0/L1 keep it bootable. All four labels are kept in sync as the pool commits—we’ll see exactly how (it’s not a single parallel write) when we get to the uberblock section.

Each label has a fixed internal layout:

The 16 KiB pad / boot envblock at the top is half padding (to avoid colliding with legacy disk labels) and half a small NVList of bootloader configuration.

The NVList is a serialized name/value dictionary describing the pool: the pool GUID, this device’s GUID, the full vdev tree topology, and a host of pool-level properties. This is also why ZFS can survive disks being renamed or moved between controllers—the kernel matches devices by GUID, not by /dev/sd? path. At import time, ZFS scans every block device it can see, reads each one’s NVList, groups them by pool GUID, and assembles the topology before doing anything else.

The uberblock ring at the bottom of each label is a 128 KiB circular buffer of fixed-size slots. The exact slot count depends on the vdev’s ashift (sector size): a 512-byte-sector disk gets 128 slots of 1 KiB each, while a modern 4K-sector disk gets 32 slots of 4 KiB. Each slot holds one snapshot of the pool’s state—a small record called an uberblock—and on every transaction commit ZFS writes a fresh uberblock into the next slot in the ring, advancing round-robin. The ring is what gives a pool its short on-disk history: at any moment its slots together hold the last 128 (or 32) commit states. We’ll dig into what’s actually inside an uberblock in its own section; for now it’s enough to know the ring exists, and that every label on every disk in the pool has its own copy.

By the end of that scan the SPA knows the whole pool’s shape and can identify every disk by GUID across reboots. But discovery is only half its runtime job—it also has to decide where on those disks every new write should land.

Space Management: Metaslabs and Space Maps

Each top-level vdev is sliced into roughly 200 metaslabs—the units of space allocation. To track free space inside one, ZFS uses a space map: a log of allocate/free operations rather than a bitmap, since bitmaps don’t scale well to multi-terabyte vdevs. At allocation time, ZFS replays that log into an in-memory tree of free ranges and serves allocations from there, so the hot path doesn’t touch disk.

When the pool is healthy ZFS allocates with a fast first-fit; once it gets nearly full it switches to a slower best-fit, and as a last resort it falls back to gang blocks (one logical block split across several smaller physical allocations). That last fallback is one reason the folklore says to keep a ZFS pool below ~80 % full.

With discovery and allocation handled, the SPA has just one job left: recording, at any given moment, where the pool’s current state actually lives on disk. That state record is a small structure at the bottom of each label called the uberblock, and it’s important enough to deserve a section all its own.

The Uberblock: The Pool’s State on Disk

The SPA, on its own, doesn’t really know what’s on the disks it manages. From its point of view, a vdev is a big array of byte ranges that some upper layer has written things into. The SPA allocates blocks, writes them, reads them back, verifies their checksums—but it has no idea whether a particular block holds a file’s bytes, a directory listing, a dataset’s metadata, or something else entirely. That’s the next layer’s problem.

But the SPA does need to record one thing for the rest of ZFS to function: at any given moment, where on disk does “the pool” actually live? Out of all those byte ranges scattered across all those vdevs, which collection is the current consistent state? That single answer is what an uberblock records—and the ring of slots we met at the bottom of each vdev label is exactly where those records get written, one slot at a time, every transaction commit.

An uberblock is small—the uberblock_t struct itself is just a few hundred bytes, padded into a 1 KiB (or larger, depending on ashift) ring slot—but it carries everything you need to find the rest of the pool. Let’s walk through what’s in it, one field at a time.

First, a magic number and a checksum. These come first because nothing else in the uberblock is trustworthy until they pass: the magic number tells ZFS “yes, this slot really does hold an uberblock, not random bytes left over from before the pool was created,” and the checksum guarantees the rest of the fields haven’t been corrupted on disk. Every read of the uberblock starts here.

Next, the transaction group number, ub_txg. A transaction group (TXG) is ZFS’s unit of atomic change: every write the kernel accepts is tagged with the number of the currently-open TXG, and roughly every five seconds (or sooner under load) that TXG is closed, all its writes flushed to disk via copy-on-write, and a fresh uberblock is stamped with the new TXG number. Since the TXG number only ever goes up, it also doubles as a global version number for the pool—“this is what the pool looked like at TXG N.” When ZFS imports a pool, it scans the rings, validates each candidate uberblock against its checksum, and picks the one with the highest ub_txg. That one—and only that one—is the active uberblock.

And finally the payload: ub_rootbp, a single block pointer (we’ll see exactly what those are in the next section) to a block somewhere on disk that holds the entire rest of the pool’s state. Every dataset, every file, every byte of metadata in the zpool is reachable by following references starting from there. The SPA itself doesn’t know what those bytes mean—it just knows where they are, which is enough for it to do its job.

Now, a subtlety in how that fresh uberblock actually lands on disk on every commit. Each top-level vdev holds four labels (L0, L1 at the head; L2, L3 at the tail), and you might assume ZFS writes the new uberblock to all four in parallel. It doesn’t—and that’s intentional. The sync goes in two passes: even labels (L0/L2) first, then a cache flush, then odd labels (L1/L3). If a crash or a torn write happens partway through, at least one pair of labels still holds an intact copy—either the new uberblock (from the first pass) or the previous one (because the second pass never started). When the second pass completes, all four rings agree, and import becomes a simple “scan any disk’s ring, pick the highest valid TXG” operation.

As we saw earlier, because the round-robin only overwrites one slot per commit, the other slots in the ring still hold previous commit states—and that’s why in ZFS you can rewind the pool a few transactions back.

That gives us the whole SPA on one picture—the pool tree on the left, the per-disk label and ring on the right, and the metaslab/space-map machinery the SPA uses to allocate inside each top-level vdev:

With the active uberblock in hand, the SPA’s job is essentially done. It knows where the pool’s state lives on disk; it just doesn’t know what to make of those bytes. To go any further we need the next layer up—the DMU—and the DMU thinks in terms of one specific structure that we keep mentioning but haven’t yet pinned down: the block pointer.

The Block Pointer: ZFS’s Building Block

The uberblock’s ub_rootbp is our first sighting of one. But block pointers aren’t unique to the uberblock—they’re the universal way every layer above the SPA addresses data. Every directory entry, every file’s content, every dataset’s metadata, every node of every internal tree: they’re all addressed by the same 128-byte structure called a block pointer (blkptr_t). If you understand block pointers, you understand most of how ZFS thinks.

A block pointer is not just an address. It carries everything needed to find the block, verify it, and interpret it:

• Up to three DVAs (Data Virtual Addresses), each pointing to a (vdev_id, offset, allocated_size) triple. The same logical block can therefore live in up to three independent physical locations.
• Logical and physical sizes (LSIZE / PSIZE / ASIZE), so the consumer knows how big the block was before compression, after compression, and after parity overhead.
• Compression algorithm (LZ4, ZSTD, GZIP, off, …) and checksum algorithm (Fletcher4, SHA-256, Edon-R, BLAKE3, …).
• A 256-bit checksum of the block’s contents. The checksum lives in the parent, not the block itself—this is what makes the whole filesystem a Merkle tree.
• Two birth TXGs: a logical_birth (the transaction group when this block was logically created) and a physical_birth (when the data at dva[0] was actually written).
• A few extra fields—object type, indirection level, and so on—that tell the reader what kind of block they’re looking at.

Because the checksum sits in the parent, every read in ZFS does the same thing: fetch the block from disk, compute its checksum, compare against the value the parent gave us. A mismatch means the data on disk has rotted, or a disk lied, or a cable flipped a bit. ZFS doesn’t shrug—it walks to the next DVA in the same block pointer (or to a redundant vdev member, in mirror/RAIDZ), verifies that copy, and silently rewrites the bad one. This is self-healing, and it falls out of the block pointer design.

So everything in ZFS, from the largest 128 KiB file block down to the smallest piece of metadata, is reachable through a chain of these 128-byte structures. The topmost link in that chain is the active uberblock’s ub_rootbp—and on the other end of it sits the DMU. Before we follow the pointer, let’s look at what kind of structures the DMU actually keeps inside the blocks it owns.

Inside the DMU: Dnodes, Object Sets, and ZAPs

The DMU exists to hide the SPA behind a single, uniform abstraction: every byte ZFS stores lives inside a typed, variable-sized, copy-on-write object, identified by a 64-bit number. Everything above the DMU—filesystems, snapshots, volumes, the pool’s own bookkeeping—is some arrangement of objects, and every object follows the same rules. Before we look at any specific piece of ZFS, then, it’s worth meeting the DMU’s three building blocks: dnodes, object sets, and ZAPs.

Dnodes

A dnode is a 512-byte record describing a single object in the DMU—roughly the equivalent of an inode in a traditional filesystem, but more general. Every object the DMU knows about is identified by a 64-bit number and described by exactly one dnode. A dnode carries:

• The object’s type (plain file, directory, ZAP, zvol, intent log, …) and the data block size for this object.
• An array of direct block pointers to the object’s data—three by default, though the exact number depends on how much room the bonus buffer takes. For small objects those pointers go straight to the data blocks. For larger objects, ZFS doesn’t try to fit a million pointers inside the dnode itself; instead, the same slots point to indirect blocks, which are blocks whose contents are themselves more block pointers. So a small file might be one hop away (dnode → data); a bigger one becomes two hops (dnode → indirect block → data); a really big one adds another level (dnode → indirect → indirect → data); and so on, growing the tree as deep as needed. ZFS labels these levels L0 for the actual data blocks, L1 for the first level of indirect blocks above them, L2 for the next, and so on.
• A bonus buffer of up to 320 bytes—free real estate inside the dnode where object-type-specific metadata lives, so ZFS can read it in the same I/O as the dnode itself.

For a regular file, the bonus buffer is where ZFS keeps the POSIX attributes—mode, uid/gid, atime/mtime/ctime, link count, and so on—so a single read of the dnode brings back both the file’s location and its metadata.

That’s all a dnode is: identity, size, on-disk location, and inline metadata, in 512 bytes. But a single dnode on its own isn’t useful—ZFS always works with whole collections of them at once, and that’s the next building block.

Object Sets

An object set (objset) is the DMU’s container for dnodes: a flat, numbered collection of them sharing a single on-disk root, written and read together. The root is a small structure called an objset_phys_t, and inside it sits exactly one thing: the dnode of object 0, called the meta-dnode. The meta-dnode’s data isn’t user data—it’s the array of dnodes for every other object in the set. So when ZFS wants object 17, it loads the meta-dnode, indexes into its data at slot 17, and gets back the dnode for object 17.

The key property is that an entire objset is reachable through a single block pointer. Hand one of those pointers to the DMU and you get the whole objset, and from there every object inside it. Filesystems are objsets. Snapshots are objsets. The MOS we’ll meet in the next section is an objset. The pattern is universal.

Dnodes and objsets get us numbered objects, but ZFS also needs named lookups—filenames, property names, well-known object pointers. That’s the third building block.

ZAPs

The third building block is the ZFS Attribute Processor (ZAP)—a single hash-based key/value object type that ZFS reuses for every name-to-value lookup it needs. Directory entries map filenames to object numbers. Pool properties map property names to values. The MOS’s object directory (we’ll see it next) maps well-known strings to object numbers. All of those are ZAPs.

ZAP comes in two flavors that ZFS picks between transparently:

• microZAP: Used when the object is small. Everything fits in a single block, key names are limited to ~50 characters, and values are 64-bit integers. The on-disk layout is a sorted array of fixed-size entries—good for small directories and configuration objects.
• fatZAP: Used when the object grows large or needs longer keys/values. fatZAP is an extendible-hash structure with a pointer table whose size doubles as collisions force splits, plus leaf blocks holding the actual entries. It scales to millions of entries while keeping lookups effectively constant-time.

This is also why ZFS has no dedicated directory structure: a directory is just an object whose dnode says “type = directory, contents = ZAP,” with the ZAP mapping filename → object number. readdir() walks the ZAP; a name lookup hashes the name and follows the table. Same machinery, reused.

One special ZAP worth flagging by name: in every filesystem objset, object 1 is the master node—a small ZAP that tells ZFS where the filesystem’s root directory lives (and a few other globals like the ZIL header object number). Whenever we land on a filesystem objset, the master node is the first thing we read; everything else hangs off the object numbers it hands us.

Putting the three pieces side by side, the DMU as a whole looks like this—an objset rooted on a meta-dnode, the meta-dnode pointing at the array of every other dnode, and one of those dnodes (a directory) holding a ZAP as its contents:

With those three building blocks in hand, we’re ready to see how ZFS uses them to express everything users actually care about.

The DSL: Datasets, Snapshots, and the Namespace

The DSL (Dataset and Snapshot Layer) is the top of the ZFS stack: everything user-facing lives here. Filesystems, volumes (zvols), snapshots, clones, the hierarchical namespace (mypool, mypool/home, mypool/home@yesterday, …), and the properties that propagate down through it (compression, recordsize, mountpoint, and so on)—all of it.

The DSL doesn’t introduce any new on-disk machinery. Every concept it manages is built out of the DMU pieces we just covered: a dataset is an object, a snapshot is an object, a clone is an object, and the relationships between them are recorded in ZAPs that map names or IDs to object numbers. Even the user-visible namespace is just a chain of ZAPs and dsl_dir objects, walked one entry at a time.

All of that bookkeeping lives in one specific object set—the very one the active uberblock’s ub_rootbp points at. It’s where every DSL concept ultimately roots, so let’s open it up.

The Meta Object Set: The Pool’s Index

Every uberblock we met in the SPA layer carries a single block pointer called ub_rootbp, and following it is how ZFS reaches the rest of the pool. The pointer sits inside the active uberblock at the bottom of every label’s ring, and it points at a special object set called the Meta Object Set (the MOS)—the index for everything else in the pool. Datasets, snapshots, the pool config, space maps, deadlists, on-disk feature flags? All objects, all living inside the MOS’s objset.

Once ZFS reads the MOS root from the uberblock, it walks into a small ZAP called the object directory, which maps well-known string names to object numbers inside the MOS:

"root_dataset"   → object N1   (the DSL_DIR for the pool root)
"config"         → object N2   (the on-disk pool config)
"free_bpobj"        → object N3   (deferred frees)
"features_for_read" → object N4
"features_for_write"→ object N5
...

Pick one of those entries—say, root_dataset—and follow it. You land in the DSL directory tree: a chain of objects that mirrors the path names you see on the command line. There’s one dsl_dir object for the pool root mypool, another for mypool/home, another for the snapshot mypool/home@yesterday, and so on—the same hierarchy you’d see in zfs list.

Each dsl_dir is paired with a dsl_dataset that holds the metadata for that one entry: creation time, snapshot lineage, how many bytes it’s using, and one more block pointer called ds_bp. That ds_bp points to another object set—a brand-new one belonging just to this dataset, separate from the MOS, where the actual files and directories live.

The MOS itself is just another object set, written via the same copy-on-write rules as everything else: when a transaction commits, the MOS root block pointer changes, and the new pointer is what ends up inside the next uberblock.

The whole DSL layer, then, is this small graph of objects living inside the MOS—the object directory naming the well-known entries, the DSL directory tree mirroring the user-visible namespace, and a dsl_dataset per filesystem and per snapshot:

That closes the on-disk loop. From the labels on every disk, through the uberblock, into the MOS, down through the DSL directory tree, into each dataset’s own objset, and out to the actual files—every step is a block pointer hop, and every commit ends with one atomic uberblock write. But on-disk structure alone wouldn’t make a usable filesystem: every read would have to touch a disk, and every fsync() would have to block for seconds. Two more pieces of machinery, sitting alongside everything we’ve covered, take care of that.

The ARC and the ZIL: Speeding Reads and Writes

Neither the ARC nor the ZIL holds canonical state. Both are optimizations layered on top of the on-disk structures we’ve just walked through, each solving a different performance problem—one for reads, one for synchronous writes.

The ARC (Adaptive Replacement Cache) is ZFS’s in-memory cache, and it’s smarter than the page cache most filesystems lean on. The ARC tracks both most recently used and most frequently used entries simultaneously, with two ghost lists recording what was recently evicted from each. On a cache miss, the ghost lists tell the ARC whether the missed block looked more “recent” or more “frequent,” and it shifts its sizing toward whichever list keeps winning. This dual-policy design copes well with mixed workloads where pure LRU would thrash.

The ZIL (ZFS Intent Log) is the answer to a different problem: synchronous writes. ZFS commits transaction groups roughly every five seconds, which is great for throughput but disastrous for an application calling fsync() and expecting durability now. The ZIL fixes this by writing a per-dataset intent log: when an fsync() arrives, ZFS records the pending operation (the intent—write, truncate, create, …) as a small log entry and writes it to disk immediately. Once that write completes, the application’s fsync() returns. The data is also sitting in the next transaction group, which will eventually flush normally. If the system crashes before the TXG commits, the ZIL is replayed per dataset on first mount after import, reapplying those operations on top of the last good uberblock.

With the on-disk layers covered and the performance machinery in place, we have everything we need to talk about the feature most people actually came to ZFS for: snapshots.

Snapshots: The Killer Feature

The same payoff that Btrfs got from copy-on-write—instant, space-efficient snapshots—is even cheaper in ZFS, and the trick comes down to a single comparison.

Creating a snapshot is O(1). The DSL allocates a new dsl_dataset object, copies the current dataset’s root block pointer into it, links it into the snapshot chain, and stamps it with the current TXG (ds_creation_txg). That’s it. No data is copied. No metadata is touched beyond a handful of objects. The snapshot now has its own root pointer to the same Merkle tree as the live dataset.

HEAD dataset  ──►  snapshot_3  ──►  snapshot_2  ──►  snapshot_1  ──►  ∅
                  (each arrow is ds_prev_snap_obj — the lineage chain)

From this moment on, the live dataset and the snapshot share the entire tree. The first time an application modifies anything, copy-on-write does its usual thing—new block, new parent, new grandparent, all the way up to a new root—but only the live dataset’s root pointer changes. The snapshot’s root keeps pointing at the old tree. Every block reachable from the snapshot’s root and not yet rewritten is shared:

The shared (green) blocks have a single physical copy on disk—both roots point at them. The grey blocks below the snapshot are the old data that the live dataset has stopped using but can’t free yet, because the snapshot still references them; they’ll sit in the live dataset’s deadlist until the snapshot is destroyed.

Now the magic: how does ZFS know whether freeing a block would lose snapshot data? In Btrfs that’s the extent tree’s reference counts. In ZFS it’s the birth TXG comparison:

if (BP_GET_LOGICAL_BIRTH(bp) > ds->ds_prev_snap_txg) {
    /* born after the last snapshot → unique to live → free now */
    free_block(bp);
} else {
    /* born before the last snapshot → shared → defer */
    dsl_deadlist_insert(&ds->ds_deadlist, bp, tx);
}

That single comparison is the heart of ZFS snapshot accounting. If a block was born after the last snapshot was taken, no snapshot can possibly reference it—it’s unique to the live dataset and can be freed immediately. If it was born before, at least one snapshot still owns it, so the block goes into the dataset’s deadlist, an on-disk record of “blocks the live dataset has stopped using but couldn’t free.” When a snapshot is eventually deleted, ZFS walks the deadlists to figure out which of those deferred blocks are now safe to release.

That’s all the moving parts. Time to put them in motion and watch what actually happens when an application reads and writes a file.

Putting It All Together: Reading and Writing a File

We’ve now seen each layer in isolation. Let’s trace two operations end-to-end, the same way we did for Btrfs—the path goes uberblock ring → uberblock → MOS → DSL directory tree → dsl_dataset → filesystem objset → file dnode → data blocks, every hop a block pointer carrying the checksum of what it points to.

Reading a File

Say we want to read /mypool/home/alice/notes.txt on a freshly imported pool.

ZFS starts at the highest-TXG uberblock from the label ring. That uberblock’s ub_rootbp gives it the MOS root. From the MOS object directory it finds the root_dataset entry, walks the DSL directory tree following mypool/home/alice, and lands at the dataset’s dsl_dataset. Inside that dsl_dataset is a block pointer to the dataset’s objset—the objset that holds Alice’s filesystem.

Inside that objset, object number 1 is the master node (a small ZAP). It points to the root directory’s object number. ZFS hashes the string "home", looks it up in the root directory’s ZAP, and gets home’s object number. It loads that dnode (which is a directory), repeats for "alice", then for "notes.txt", and finally has a dnode for the file itself.

Now the dnode’s direct (and possibly indirect) block pointers describe the file’s contents. ZFS computes which logical offset the read covers, follows the block pointer chain down to the right L0 data block, and asks the ARC for it.

If it’s an ARC hit, the data comes back from RAM in microseconds. If it’s a miss, the request goes into the ZIO pipeline: a staged async pipeline that resolves the DVA to a vdev and offset, issues the physical read, decompresses the result if compression is on, and finally verifies the checksum against the value the parent block pointer carries. If the checksum doesn’t match, ZIO transparently retries against the next DVA (or another mirror member, or RAIDZ parity), and once a good copy is found, ZFS schedules a write to repair the bad one. The application sees nothing but correct data.

Reads are the easy direction—nothing on disk changes, so the worst that can happen is a checksum miss. Writes are where copy-on-write earns its keep.

Writing a File

When the application calls write(), the new bytes go into the page cache and the affected dnode is marked dirty in the current open transaction group. There is always one open TXG accepting writes; new TXGs open as old ones close, on a rolling cycle.

Roughly every five seconds, the open TXG closes and starts syncing:

1. ZFS picks new physical locations for the dirty data blocks (via metaslab allocation).
2. The new data blocks are written to their new homes through the ZIO pipeline—compressed, checksummed, and possibly mirrored to multiple DVAs.
3. The L1 indirect blocks above them are updated to point at the new L0 locations and written to their own new locations.
4. This propagates up the tree: new dnode block, new objset root, new MOS, until finally a brand new uberblock is constructed pointing at the new MOS root.
5. The uberblock is written to its slot in the ring on every label of every active vdev, with cache-flushing barriers around the write.

That last write is the commit point: before it, the pool’s official state is still the previous uberblock; after it, the whole batch is atomically live. If power is lost in between, the new blocks become unreferenced garbage and the previous uberblock is still the one that imports.

Synchronous writes that landed in the ZIL during the TXG are already durable, as we saw in the ARC/ZIL section—the ZIL is replayed on next mount and reapplied on top of whatever uberblock won.

Summary

ZFS rests on four ideas: a 128-byte self-describing block pointer that turns the pool into a Merkle tree; a layered architecture (SPA / DMU / DSL); copy-on-write that commits whole transaction groups atomically by stamping a fresh uberblock; and birth TXGs that make snapshots O(1) and snapshot accounting a single comparison.

The SPA arranges disks into a tree of vdevs and anchors each disk with four redundant labels, each carrying the pool’s NVList and an uberblock ring. The DMU turns blocks into typed, copy-on-write objects described by dnodes, packed inline under the meta-dnode. The DSL sits on top: the active uberblock points at the MOS, whose object directory leads down through the dataset tree to the filesystem objsets where files actually live. Snapshots are a single TXG-stamped dsl_dataset linked into the chain; deadlists track shared blocks until the snapshot is destroyed.

Alongside the stack, the ARC caches blocks with an MRU+MFU policy, and the ZIL logs fsync() intents so synchronous writes don’t have to wait for the next TXG commit. Everything goes live with one uberblock write—if it didn’t land, the previous tree still is the live tree.

And with that, I’m closing the Filesystems series for now. We’ve walked from FAT32 and ext4 through XFS and NTFS, into the copy-on-write world of Btrfs, and finally through ZFS’s three-layer architecture. There’s plenty more out there—APFS, bcachefs, the log-structured ones, the distributed and network filesystems—and maybe in the future we’ll come back to them, but this is a good place to pause. Thanks for reading along.

Want to dig into the source? OpenZFS lives at github.com/openzfs/zfs . Start with include/sys/spa.h and include/sys/uberblock_impl.h for the on-disk structures, module/zfs/vdev_label.c for label and uberblock handling, module/zfs/dmu.c and module/zfs/dnode.c for the object layer, module/zfs/dsl_dataset.c for snapshots and clones, module/zfs/metaslab.c for allocation, module/zfs/zio.c for the I/O pipeline, module/zfs/arc.c for the cache, and module/zfs/zil.c for the intent log. Happy reading!