Deep Dive 2: The Technical Requirements for Liquid Training

The world has seen that distributed training is possible. [1] But alas, the journey does not end here.

The question worth asking now sits one layer above: can a system tolerate nodes constantly joining, leaving and restarting without quietly degrading the model or destroying the run or eroding the very cost advantage that made the distributed compute worth using in the first place?

Conventional synchronous infrastructure and scheduling is, at heart, a bin-packing exercise.

Devices are wired tightly together and work is scheduled to keep step latency as low as possible, which makes the arrangement fast but also rigid: it quietly assumes a fixed set of workers, uniform hardware, highly reliable devices, and a fast interconnect holding everything in place.

It is a good instinct, but it only holds if that lower layer offers firm guarantees, because the moment it does not, the hidden complexity resurfaces as a kind of cognitive debt the builder eventually has to pay back.

There is a ladder implied here, from an isolated single GPU, to distributed but uncoupled workloads, to coupled ones, and finally to the tightly coupled distributed training iota is built for, and each rung upward asks more of the coordination layer than the last.

Take that architecture and set it on top of capacity that moves, differs in speed and memory, and talks over the open internet, and its assumptions do not bend so much as break, all at once and together.

Liquid training starts from the opposite premise. It treats change in participation and available hardware and bandwidth not as a fault to be recovered from, but as the norm.

The first thing liquid training has to tolerate is interruption. Capacity needs to be able to join a run, pause, drop out entirely, and rejoin later, with the run carrying on from where it left off rather than starting again.

This is precisely where conventional training is most brittle.

A synchronous system assumes a largely fixed roster of workers, so when one disappears it can leave with an accumulated model and optimiser state in hand, and the run either stalls while the topology is rebuilt or simply loses the work that device was carrying.

It is also why the useful unit of work has to become smaller: when progress is locked into large, indivisible chunks, only long and stable windows of compute can take part, and most real-world capacity is neither.

The mechanism that addresses this is checkpointing, working at two levels.

• Periodic local checkpointing bounds how much progress is lost when a device drops out, by writing recent state to a checkpoint a replacement can resume from.

• Network-level checkpointing lets a replacement initialise directly from the current model and optimiser state rather than from nothing.

Together they turn training from a fixed-topology problem into something closer to an asynchronous, fault-tolerant one, and that shift, more than any single trick, is what separates liquid training from distributed training as it is usually understood.

The same logic runs in reverse: joining is not free either.

Every new device introduces a perturbation while it comes up to speed, and because that perturbation grows super-linearly when several arrive at once, it is almost always better to onboard capacity gradually, one device at a time where possible, than in a flood.

None of this makes interruption costless, and it would be dishonest to imply otherwise, but it does make the cost measurable, and small enough to plan around.

We think of it as a churn tax: the additional tokens a training run must process to recover its trajectory after each change in participation. Modelled simply, as a one-time penalty per event, it reduces to a ratio anyone can reason about:

With this simple model we can predict the impact of N churn events, each costing dT additional tokens, over a total budget of T. A run that sees a hundred such events, each costing ten million tokens in ‘tax’, across a fifty-billion-token run, carries an overhead of roughly 2.0% in time and cost.

The second requirement is that a run can draw on mixed hardware; different generations, memory profiles, and speeds, rather than the matched, uniform fleet conventional training quietly depends on.

Orion-16B is currently training across c.180+ heterogeneous GPUs, a mix of RTX 4090s and 5090s, coordinated as a single run. Whether mixed hardware can be made to work is, at this point, a question we are answering in public.

The reason it matters is commercial more than it is technical. Matched fleets of current-generation accelerators are the most contested and expensive resource on the market, so a system that can only run on them is fishing in the smallest, priciest pool there is. Real installed capacity looks nothing like that clean picture; it is a patchwork, assembled from whatever happens to be free, and a naïve run over it moves at the pace of its slowest contributor, leaving faster machines idle.

The mechanism is to schedule around a single figure of merit, useful throughput per dollar, TFLOPS/$, rather than fleet-level utilisation.

For Orion-16B we paired 4090s and 5090s specifically to bring their throughput into approximate balance, keeping the pipeline flowing and lifting overall MFU. But the same logic can cut the other way: we will deliberately underutilise a device where doing so strictly improves TFLOPS/$.

Utilisation is not the goal, useful work per dollar is, and the two are not always the same. It is the same figure that prices interruptibility’s churn tax, and it is not hard to imagine eventually surfacing to the buyer as a single dial between time and cost.

Buyers prioritise, in order: an equivalent-quality model first, a familiar training interface — torch, an SDK, the standard tooling — second, comparable speed within a workable margin third, and raw cost fourth.

Heterogeneity tolerance is what lets a system satisfy all four without forcing anyone onto matched, expensive hardware. And there is headroom still to come: active research by the iota team aims to place larger shards, or larger batches, on more capable devices, so each machine carries a workload scaled to its capacity rather than levelled down to the weakest, sharpening the same economics, and uniquely available to a system built around heterogeneity from the outset.

Orion-16B is a live demonstration that a mixed fleet holds together in one coherent run, and the Pluralis models point the same way [4]. The cost model is sound, the assignment problem tractable, and the approach now being proven in the open rather than argued from first principles.

The third requirement is that a run can keep making progress when its machines are far apart and joined by ordinary internet links rather than datacentre-grade interconnect.

In a centralized cluster, you can hide the cost of shuffling data behind a super-fast local network, but if you spread that same training run across different regions, every step in the process ends up waiting on data crawling over much slower, less reliable links.

Left alone, the result is machines sitting idle waiting on the network, and the cost advantage that justified going distributed in the first place quietly evaporates. The answer is not necessarily faster links but to send less across them.

There is real precedent for this. SWARM parallelism has shown that training can survive nodes joining and dropping out across unreliable networks [5]; low-communication data-parallel methods such as DiLoCo and DeMo cut the synchronisation overhead [6] between workers by orders of magnitude [7], and decoupled variants of DiLoCo add resilience to worker failure on top of that [8].

The harder frontier, pipeline parallelism across low-bandwidth links, is the one our own work on Residual Bottleneck Models takes on directly [9], compressing the volume of data exchanged at each step rather than assuming a wider pipe.

And the clearest evidence that the approach holds is not a diagram on a whiteboard but Orion: a real model, trained across real distance.

For Orion, we began with an early pretraining proof of feasibility: a 100-billion-parameter model trained using pipeline parallelism across single GPUs spread over several datacentres, coordinated over ordinary internet links, with Residual Bottleneck Model compression carrying the communication load. It reached roughly 65% of the training speed of an equivalent datacentre setup, on hardware costing a fraction as much.

That result validates the bandwidth argument at a scale distributed training had not previously reached. And the picture is corroborated further down the parameter range: in controlled comparisons, iota tracks a centralised baseline closely, converging to it within a few billion tokens, with fault tolerance, model splitting, and internet-scale coordination adding no meaningful machine-learning penalty.

The figures below show iota producing models equivalent to DiLoCo’s and converging to the centralised baseline, evidence that activation compression does not damage the model.

That gives us two results that mark the ground already covered:

• Orion-100B — efficient hundred-billion-parameter training on globally distributed commodity hardware.

• Orion-16B — efficient heterogeneous decentralised training across hundreds of GPUs.

The first proved the approach holds at scale; the second, live now, proves it holds across mixed hardware. Between them they answer the two questions that matter most: can a model this large be trained this way at all, and can it be done on the imperfect, varied fleet the real world actually offers.

What remains is the widest version of the problem: less controlled, more open participation, where the interruptibility and heterogeneity arguments still lean more on mechanism and testing than on a finished, demonstrated run, that is the next thing to prove.

The world is not short of installed GPUs. What it is short of is a way to turn imperfect, scattered, intermittent supply into useful training, at a cost and reliability a buyer can plan a roadmap around. That is the system iota is being built to be, and Orion is the first real evidence that it works.

• Read the Introduction Thesis → The Case for Liquid Training

• Read the Economic Thesis → The Economics of Liquid Training

• Distributed pre-training arrives → Orion-100B and our results

• See iota in action → Orion 16B Live Run Dashboard

• Learn about the Macrocosmos ecosystem → Macrocosmos Home Page

Citations referenced:

[1] Erdil, E., & Barnett, J. How far can decentralized training over the internet scale? Epoch AI, Gradient Updates. https://epoch.ai/gradient-updates/how-far-can-decentralized-training-over-the-internet-scale

[2] Quinque, F., et al. (2025). Incentivised Orchestrated Training Architecture (IOTA): A Technical Primer for Release. Macrocosmos AI. arXiv:2507.17766.

[3] The Economics of Liquid Compute. Macrocosmos AI. macrocosmosai.substack.com/p/the-economics-of-liquid-training

[4] Ramasinghe, S., Ajanthan, T., Avraham, G., Zuo, Y., & Long, A. (2025). Protocol Models: Scaling Decentralized Training with Communication-Efficient Model Parallelism. Pluralis Research. arXiv:2506.01260.

[5] Ryabinin, M., Dettmers, T., Diskin, M., & Borzunov, A. (2023). SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient. Proceedings of the 40th International Conference on Machine Learning (ICML). arXiv:2301.11913.

[6] Douillard, A., Feng, Q., Rusu, A. A., Chhaparia, R., Donchev, Y., Kuncoro, A., Ranzato, M., Szlam, A., & Shen, J. (2023). DiLoCo: Distributed Low-Communication Training of Language Models. Google DeepMind. arXiv:2311.08105.

[7] Peng, B., Quesnelle, J., & Kingma, D. P. (2024). DeMo: Decoupled Momentum Optimization. arXiv:2411.19870.

[8] Douillard, A., Rush, K., Donchev, Y., Charles, Z., Fallen, N., Dubey, A., Gog, I., Dean, J., Woodworth, B., Garrett, Z., Keating, N., Bishop, J., Prior, H., Yvinec, E., Szlam, A., Ranzato, M., & Dean, J. (2026). Decoupled DiLoCo for Resilient Distributed Pre-training. Google DeepMind. arXiv:2604.21428.

[9] Aboudib, A., Lopez Portillo A., R., Brady, K., & Cruz, S. (2026). ResBM: Residual Bottleneck Models for Low-Bandwidth Pipeline Parallelism. Macrocosmos AI. arXiv:2604.11947.