Diving deep into systems
What if the database had no idea what your WHERE clause clause actually meant until it had already dragged every single row off disk into memory? That used to be the default, ages ago. The storage layer was just a dumb block server handing your scan operator the pages to
I've been thinking about what the bottom of a database should look like if we stop asking it to understand the database. That sounds a bit backwards, so let me be more specific. Most databases make an early decision about the shape of the world. Rows belong to
We've built one Constellation: a few tens of thousands of machines, a ledger, a flock of schedulers, an autoscaler, identity, the works. Now you have more than one, because you have more than one region, or you've grown past what a single failure domain should hold.
Way back in the object-model post I made a point of saying orbit-core has no I/O, no network, no disk, just types and the functions over them, and in the ledger post I kept calling it an append-only, replayable log. Those weren't aesthetic preferences.
I'll just assume if you're here, you've read Jared and Andrew's posts, if not there are the links. If not, honestly just close this post and don't waste your time. The last couple of days have seen a very public
Config is where most people actually meet an orchestrator, and Kubernetes greets them with YAML. Then, because YAML can't express anything dynamic, it greets them with templated YAML, which is how the world got Helm and Kustomize and a working knowledge of where whitespace is significant. Borg had
This is the post where "improve on Borg today " stops being a slogan. Borg was designed for a world of fungible CPU and memory, and that world is gone for an enormous and growing slice of what data centers actually do, which is train large models. Almost every
Networking is the one place Orbit refuses to take Borg's advice. Borg's own retrospective says to give every unit its own IP, and Kubernetes did exactly that, an IP per pod, and then spent a decade building CNI plugins, overlay networks, IPAM to hand out the
The reclamation post ended on a debt: the whole utilization story depends on the reservation being a good estimate of real usage, and nobody said where that estimate comes from. The answer is a control loop, and the deeper answer is a position I'll defend for the rest
Here's a number that should bother you: most clusters run at something like a third of their capacity, and the operators know it, and they pay for the other two thirds anyway. The reason is that everyone sizes their workloads for the worst case. You ask for the
Everything I've written about so far is the control plane: the ledger, the schedulers, the airlock. Now go stand next to an actual machine in an actual rack. On that machine runs the Satellite, the per-node agent, and it has one property that matters more than all
Every system needs a front door, and the front door is where you decide what "allowed in" means before anything inside has to care. In Orbit that's the Airlock. Nothing enters a Constellation without passing it: the client proves who they are, the mission proves it&
A scheduler is where an orchestrator earns its keep, and it's also where Borg admitted it had painted itself into a corner. The scheduling policy lived in one place, it grew over years to serve every workload Google had, and the paper is candid that the single scheduler
The telemetry plane is the easy half of the state split to describe and the easy half to get quietly wrong. It carries everything the cluster observes about itself: usage, health, pressure, liveness, GPU temperature. It's high-volume, it's allowed to be stale, and the entire
The Ephemeris is the part of Orbit I was most nervous about, because it's the part that has to be correct. Telemetry can drop a sample and shrug. The ledger cannot drop a placement, can't double-book a node, and can't disagree with itself