Data Movement Optimization in Disaggregated Memory Systems

Disaggregated memory breaks the assumption that has held since the earliest von Neumann machines: that memory lives at a fixed, short distance from the CPU that uses it. With CXL (Compute Express Link) Type 3 memory expanders, memory pooling appliances, and fabric-attached memory, a page of RAM a process touches might sit several switch hops away, reachable only through a coherent but higher-latency fabric rather than the local DDR channel.

The kernel already has a name for “memory that isn’t equally close to every CPU” — NUMA. What disaggregation adds is degree: latencies and bandwidth asymmetries an order of magnitude larger than classic multi-socket NUMA, plus the reality that some of this memory has no CPU attached to it at all (a bare memory expander is a NUMA node with has_cpu unset). The kernel represents this today, and you can see it directly:

$ ls /sys/devices/system/node/node0/
has_cpu  has_generic_initiator  has_memory  has_normal_memory  ...

Once memory is disaggregated, moving data between tiers stops being an occasional NUMA-balance nicety and becomes the central performance problem. This article is about that movement: how the kernel exposes migration primitives (move_pages(2), migrate_pages(), HMM), how DMA engines execute the underlying copy, and — this is the part textbooks skip — why the batching strategy around that copy dominates achievable bandwidth far more than the copy itself does. We built and measured a software model of a batching DMA engine to make the effect concrete, not hypothetical.