Glossary
220 phrases the manifesto and its companion lean on, decoded.
Preamble
build correctness into the shape of the system, not into the vigilance of the people keeping it going
Make correct behaviour a structural property of the system - something encoded in types, schemas, atomic operations, ownership and boundaries - so that it holds whether or not anyone remembers a rule or runs a check by hand. This is the manifesto's central thesis. Every rule that lives only in memory gets forgotten when a deadline hits, and every defence you have to run by hand gets skipped, so correctness has to live somewhere it cannot be skipped, not in the diligence of whoever happens to be at the keyboard right now. It reads like motivational advice about writing careful code or building a good culture, but it is actually the opposite. It tells you to stop leaning on care, and instead to change the shape of the thing so that the careless path just isn't there to take.
minimise what a tired engineer has to hold in their head to make a correct change, while keeping the blast radius bounded for anything an attacker or an unlucky caller controls
This is the manifesto's actual objective function, and it is written on purpose in the form minimise X subject to constraint Y. The thing you are minimising is the cognitive load a safe change demands, meaning how much an engineer has to understand and keep straight to change the system without breaking it. The constraint you must never break is the damage that inputs you don't control can do: an attacker or an unlucky caller must not be able to cause wide harm. Writing it as a two-term optimisation rather than a list of nice-to-haves matters for two reasons. Every later tension in the document resolves back to this objective and not to a slogan, and it makes plain that the second term is a constraint, not something you trade away when it's convenient.
blast radius
The blast radius is the set of components, data, and users that can be harmed when one component fails or gets compromised - how far the damage spreads before something stops it. The term is borrowed from cell-isolation and fault-isolation work, and it turns the vague question "how bad is a failure" into a concrete quantity you can bound and design against. In the manifesto it is the second term of the objective. You keep what a tired engineer has to hold in their head small, subject to keeping the blast radius bounded for anything an attacker or unlucky caller controls. The whole point is that a small blast radius is a property you architect ahead of time, through cells, bulkheads, isolation, and not damage you go and measure after the fact.
Most of the tenets here buy you that first term. The ones that add code and state to contain failure are paying for the second.
The objective in the preamble has two terms: minimise what a tired engineer must hold in their head to make a correct change, and keep the blast radius bounded for anything an attacker or unlucky caller controls. This line treats those two terms as a budget. Most tenets spend their effort on the first term and cut cognitive load. A smaller set deliberately adds code and runtime state - queues, breakers, retries, validation - to contain failure, and that extra complexity is the price of the second term. The point is that no tenet is free. Even the failure-containment ones cost you something, so you work out which term a given rule is paying for before you decide it's worth it.
If the input is caller- or attacker-controlled and the blast radius is wide, pay the cost now. If it's self-controlled and contained, you can defer it, but write down why.
This is the concrete tie-breaker for any conflict between tenets, and it comes straight out of the two-term objective. If untrusted input could cause damage across a wide surface, build the defence now, because the cost of skipping it is unbounded and out of your hands. If the input is your own and the consequences stay contained, you're allowed to defer the work, but you have to record the reasoning so the deferral is a deliberate decision you can revisit later rather than a silent gap. It takes what could read as a vague cost-benefit hand-wave and turns it into a decision rule keyed on exactly two variables, which are who controls the input and how wide the blast radius is.
None is free
A list of best-practice tenets reads like good advice you can apply everywhere, but every tenet here carries a cost and sits in tension with the others, so applying any one of them blindly trades against another. That's why each tenet names both the failure it prevents and the tension it creates instead of presenting itself as a free win. The line is there to head off cargo-culting. These are judgement calls with prices attached, not rules you follow on autopilot.
I. Locality of reasoning over global cleverness
action-at-a-distance
Action-at-a-distance is code whose behaviour gets silently changed by state set somewhere far away - a flag passed into a constructor, a mutated global, an environment variable read deep in some call. You can't understand code like that by reading it on its own, because the thing that decides what it does isn't in front of you. The name borrows the physics image of one body affecting another with nothing visible in between, and it pins down the exact cost that locality fights against. Before any edit is safe, the reader first has to reconstruct invisible context they had no way of knowing was load-bearing.
You've written a puzzle and scattered the pieces across the repo
When a function's correctness depends on the call order of other functions, and a flag set in a constructor, and a global mutated somewhere else, the logic you need to understand that one unit isn't in front of you. It's fragmented across places you can't see from where you're reading. The reader has to hunt through the whole codebase and reconstruct that invisible context before they can safely touch anything. This is the cost of action-at-a-distance, and it is the largest tax on changing code. The opposite of it, locality, is what lets someone make a correct change while staying ignorant of the other ninety-nine percent of the system.
II. Make the data flow explicit; ambient dependency one override point
seam
A seam is a single declared place where you can substitute behaviour - injecting a clock, say, or overriding a dependency - without editing the code under test. It's Michael Feathers' term from working with legacy code, and it's the precondition for honest tests. Ambient dependencies like the wall clock or a module-level network client stay untestable until there's one deliberate override point where a test can slot in a fake. Calling it a "seam" rather than a vague "joint" matters, because the value is exactly that it's the one designed substitution point and not an accidental gap.
a flake you'll chase for days
A flake is a flaky test, one that passes and fails non-deterministically on the very same code, so a green run proves nothing and a red one might not be a real bug. Here the cause is an ambient dependency such as a hardcoded clock or a module-level client, which makes the test's outcome turn on the time of day or on shared state rather than on the code under test. The phrase reads to a newcomer like an unreliable person you keep pursuing, and that misread is apt. You'll burn days chasing a failure that vanishes the moment you look at it, which is exactly the concrete pain that injecting a seam takes away.
the tests need a séance to reproduce anything
When behaviour depends on ambient state like now() or random() pulled from the ether, the same inputs give you different outputs, so you can't deterministically re-create a failing run. You're left guessing at conditions you have no way to summon when you want them, a bit like trying to contact the dead. That image casts non-reproducibility as occult guesswork, which is the opposite of the controlled, injectable seam the tenet is after. The fix isn't to ban ambient dependencies. It's to give each one a single declared seam a test can reach, so reproducing a run comes down to control and not luck.
parameter soup
This is the mess you end up with when you over-apply the naive reading of dependency injection, threading a clock or an RNG explicitly through forty layers of function signatures until every call is bloated with plumbing. The tenet's rule was never pass everything as an argument. It's that every ambient dependency has exactly one declared seam a test can reach, say an injected default, or an overridable context, or a virtual clock. Parameter soup is the name for the failure mode the single-seam rule exists to head off, so you neither bury dependencies invisibly nor smear them across every signature.
III. Parse, don't validate: make the illegal state unrepresentable
Parse, don't validate
Instead of checking input and then throwing away the proof that you checked it (validate), you transform the input once into a narrower type that carries its own validity (parse), so the unchecked form just can't be held downstream. Alexis King's coinage makes the distinction sharp. A validating function hands back the same loose type it received, so every later call site has to wonder "has this been checked?" and re-ask, and sooner or later someone forgets. A parsing function returns a type that can only exist if the check passed, so checked-ness is permanent and you can't forge it. The two words aren't interchangeable, and it isn't telling you to skip a step. It tells you to make the one check produce evidence the type system then carries around for you.
make the illegal state unrepresentable
Design your types so that a contradictory or nonsensical combination, like being loading and loaded at the same time, literally can't be constructed or compiled, rather than getting constructed and then rejected by some runtime check. Yaron Minsky's coinage moves enforcement off human vigilance and onto the type system. A whole class of bugs disappears once, for every call site that will ever exist, instead of being caught (or missed) one defensive check at a time. This isn't impossible perfectionism and it isn't a runtime guard. It's choosing a representation where the bad states have no spelling, which is also why the tenet warns it can be overdone.
discriminated union
A discriminated union is a type that is exactly one of several tagged variants at a time, for instance loading or error or loaded, never a mix and never none of them. The tag is what lets the compiler force you to handle each case, and it stops you reading fields that only exist in some other variant. It's the practical tool for making illegal states unrepresentable. It swaps out a cluster of independent booleans (which can encode contradictions like a spinner showing while stale data is also sitting there) for a single value whose nonsense combinations are simply untypeable.
phantom types
Phantom types are type parameters that carry compile-time information, such as a "validated" marker, with no matching runtime representation at all. The tag exists only to the compiler and costs nothing when the code runs. They let you encode facts like "this string has been sanitised" into the type, so the compiler refuses to mix sanitised and unsanitised values. The manifesto points to them as the place the unrepresentable-state idea is most easily taken too far, into "a cathedral of phantom types nobody can read", where the type-level machinery ends up harder to maintain than the bug it was meant to prevent.
a whole cathedral of phantom types that nobody can read
Phantom types are compile-only marker types that carry no runtime data but let the type system enforce invariants. This phrase is a warning that the tenet's own advice, make illegal states unrepresentable, can be pushed to a grand, over-engineered extreme where the type machinery gets so elaborate it obscures things instead of clarifying them. Past that point the cleverness is its own complexity tax and it defeats the locality it was supposed to serve. The corrective is to knock the principle back down to a single chokepoint everything funnels through, rather than an ever-taller edifice of types.
the intern at a psql prompt
This stands in for every writer you didn't anticipate, right down to the least careful, lowest-context one hitting the database directly. The whole point of a schema constraint (NOT NULL, CHECK, a foreign key, a partial unique index) is that it refuses the bad row no matter who issues the write, the migration script and the careless newcomer at a raw SQL prompt included. Structural constraints protect against actors that no code review ever covers, because the database itself enforces the invariant instead of relying on every caller to remember to.
it still beats forty scattered checks, which beat a comment
This ranks three levels of enforcement by structural strength. A single validation at the edge, where untrusted data comes in, is best, because it establishes the invariant once, in a place nothing can bypass. Lots of runtime checks scattered through the code are worse - they're repetitive, it's easy to miss one, and the fact that you needed forty of them proves the invariant was never really established - but they're still real, executable enforcement that fails when it's violated. A comment is worst of the lot, because prose enforces nothing. Nobody runs it and nothing breaks when it gets ignored. The point is that executable structure beats repetition, and repetition beats documentation with no teeth.
IV. Everything across a trust boundary is hostile until proven otherwise
trust boundary
A trust boundary is any point where control, data, or authority crosses from a party you don't control into one you do: a socket, a form field, a queue, a third-party dependency, a CI pipeline. The term comes from threat modelling, and what makes it useful is that it marks exactly where input has to stop being assumed friendly and start being treated as hostile and re-checked. It swaps a fuzzy sense of "code I trust" for a concrete line on the system. Attacks come in at these crossings, so this is where parsing, validation, and limits belong.
crashing on malformed input is a free DoS
Crashing on bad input feels like virtuous fail-fast behaviour, but if some well-formed-but-nasty value can kill your process, then an attacker gets a denial of service for the price of one request. The reframe turns what looks like a robustness bug into a security hole. Robustness at a trust boundary means surviving hostile input, not dying loudly on it. The cheaper the crash is to trigger, the cheaper the attack.
the 2 GB JSON body that parses fine and then kills your heap
The comforting intuition is that if input parses cleanly it must be safe, and the real danger sits with malformed data. But a well-formed payload of attacker-chosen size sails straight through validation and then exhausts memory, because you allocated proportional to a size the attacker controls. Well-formedness bounds shape, not magnitude, so a perfectly valid structure can still be a denial of service. The fix is to bound size explicitly at the boundary, separately from validating shape.
billion laughs
The billion laughs attack is a named XML entity-expansion denial of service where a tiny document expands exponentially into gigabytes. A handful of nested "lol" entities, each one referencing the one below it, blow up into billions of characters once the parser dereferences them. The whimsical name hides a serious resource-exhaustion bug, and it's the canonical small-input/huge-expansion bomb behind the advice to bound output size before you allocate, rather than trusting that a small input means small work. It generalises well beyond XML, to any decompression, expansion, or recursion driven by attacker-controlled input.
NaN-poisoning a model you only discover three weeks later
A single not-a-number value, once it gets into a computation, spreads through everything downstream without a sound, because NaN arithmetic just gives you back NaN. So it quietly corrupts a model's output, and the damage doesn't surface until weeks later, somewhere far from where it started. That poisoning image is about how one unclamped float infects a whole pipeline instead of failing loudly right at the source. Which is the reason you clamp and validate at the boundary, at ingestion, rather than trying to chase the corruption down long afterwards and a long way from where it actually went wrong.
A typosquat or a poisoned transitive dep is a boundary crossing, not a convenience
Installing a package feels like a developer convenience, and your own dependencies feel like trusted inside code. But a malicious lookalike package, or some compromised indirect dependency, is untrusted input coming into your system, and it deserves the same scrutiny you'd give a network request. The reframe makes dependency management a trust-boundary problem. Code you shipped but didn't write runs with your full authority, so the supply chain is an attack surface and not just a checkout step.
the code you didn't write but shipped is still code you shipped
Third-party dependencies, build plugins, CI tooling - all of it runs with your privileges, your secrets and your keys, which means it sits inside your trust boundary even though you never wrote a line of it. So importing code is really just accepting input, and that input is hostile until proven otherwise. There's no privilege boundary around an in-process import the way there is around a network call. Treating a dependency as harmless because it happens to be popular is the mistake. You're still on the hook for everything that ships under your name, so dependencies and tooling have to be pinned, verified and scoped just like anything else inside the boundary.
client-side validation is UX, not security
Because the browser form already turns away bad input, it feels like the input has been checked. But a client-side check only smooths things over for the honest user by giving fast feedback. An attacker skips the browser completely and talks straight to your server, so the only check that actually protects you is the one running server-side at the trust boundary. The check the user sees is not the check that defends you, and it has to be done again where the request really lands.
Authentication tells you who; it says nothing about what they may do
Authentication and authorisation are two separate checks, and they're easy to mix up because they sit right next to each other and even sound alike. Authentication establishes identity. It proves who is making the request. Authorisation is a different decision altogether, about whether that identity is allowed to do this particular thing to this particular resource. The common and dangerous misreading is that being logged in means being allowed, and that's exactly the assumption behind the classic privilege-escalation bug, where a perfectly valid user reaches an action they were never meant to have. The two have to be checked on their own, because knowing who someone is tells you nothing about what they're permitted to do.
zero-trust between your own in-process functions buys you clutter for no real reduction in threat
If zero-trust is good, then applying it everywhere, even between your own functions, sounds even safer. But re-validating at every internal hop just adds noise and latency and reduces no real threat. Zero-trust is a discipline for boundaries, not for every single call site. Trust the parsed type as it flows inward, and only re-parse at genuine boundaries. Misapplied internally it falls apart into the scattered re-checking anti-pattern that parsing once at the edge was supposed to get rid of.
V. Keep the responsive path free of uncontrolled-latency work
backpressure
Backpressure is an explicit signal sent back to a producer saying the consumer is full and can't take any more - returning a 503, say, or a "queue full" rejection instead of quietly accepting the work. The term is borrowed from fluid mechanics, and it's how a bounded queue protects itself. Rather than buffering away without limit until memory runs out, it pushes the overload signal back upstream so the producer slows down, sheds, or fails fast. The key shift here is making fullness a proper response that the caller has to handle, instead of an invisible build-up that collapses the whole system eventually.
freshest-first
Freshest-first means that under overload you serve the newest requests first, because the oldest ones have most likely already blown past their deadline, and answering them just wastes effort on results nobody can use any more. It's a deliberate inversion of the usual FIFO intuition, where doing the oldest item first feels like the only fair thing. When work expires, though, fairness is the wrong goal. Draining stale work first means you burn your scarce capacity producing dead answers, whereas serving fresh requests keeps the most still-useful work moving.
deadline-aware shedding
Deadline-aware shedding is fast-rejecting work whose deadline will expire before you could possibly finish it, rather than dutifully grinding through the work and then throwing away a result that turned up too late. It's a refinement of plain load shedding, which just drops some fraction of the load. Here the choice of what to drop is informed by each item's deadline, so you drop precisely the work that was doomed. Killing hopeless work early frees up capacity for requests that can still be served in time, and that's what keeps a system useful under overload rather than just busy.
each one is a heartbeat with a deadline it owes
A UI thread, an event loop, a request handler, a game frame, a scheduler tick - each one has to keep ticking on a regular basis within a time budget it owes to someone or something. A user's eye, a watchdog, a sixteen millisecond frame deadline. The phrase puts together the rhythmic must-not-stop quality of a pulse and that hard time budget. If the responsive path synchronously blocks on slow work, the beat gets missed, and the system looks dead even though it's only waiting. Which is why the general move is to get slow work off the responsive path and make it observable, so the one thread a human is watching never goes dark.
its responsiveness is hostage to a stranger's worst day
The moment your responsive path synchronously waits on a slow network, or a disk, or a lock held by who-knows, your latency stops being yours to control. It's now dictated by some outside party's outage or overload or bad day. Hostage names the loss of control. A stranger's worst day names the fact that the failure starts entirely outside your system, in code and conditions you don't own. That's the reason for moving such work off the responsive path - with async and a pending state, say, or a job queue and a 202 - so an outside party's failure can't freeze the thread a user is actually watching.
the queue is just one instance of the rule, it isn't the rule
The underlying principle is to move work with uncontrolled latency off the responsive path and make that work observable. A bounded queue is one concrete, domain-specific way to do that, but it's only an instance of the rule and not the rule itself. The warning is against cargo-culting the mechanism. If you fixate on the queue, you'll miss that batching, a worker pool, a deadline scheduler or a circuit breaker can each be the right way to make the same move in a different context. Learn the abstraction, which is get slow work off the hot path and watch it, and the queue turns into just one tool you might reach for.
a deep buffer just delays the rejection and fills up with already-dead work
Intuition says a bigger buffer helps, because it absorbs more load, and that intuition is right for short bursts but wrong for sustained overload. By Little's law, queues smooth out bursts, they don't add throughput, so under continuous overload arrivals outpace service no matter how deep the buffer goes. A large queue then does two bad things at the same time. It postpones the rejection that's going to happen anyway, and it fills up with items that have already blown past their deadline, so by the time you get to them the work is dead and serving it is just wasted effort. The fix isn't a deeper buffer. It's deadline-aware shedding, and processing the freshest work first.
VI. Every wait across an uncontrolled boundary has a deadline
a deadline
A deadline is an absolute point in time by which a cross-boundary wait has to end. Ideally you push it through the whole call chain as a shared budget instead of starting a fresh fixed timeout at every hop. That difference from a naive timeout is really the whole point. Set a per-call timeout of five seconds at three layers and you can end up waiting fifteen seconds in total, but a propagated deadline says "everyone downstream must be done by this clock time" and it keeps shrinking as time gets used up. So it takes an unbounded hang, the kind that spreads by eating the caller's threads and then their caller's threads, and turns it into a bounded error you can see, one that fails at a moment you can predict.
a hung dependency without a deadline doesn't fail, it spreads
A hang sounds like one call stalling in one place. But with no deadline that stalled call holds onto the caller's thread, and that exhausts the caller's pool, which stalls its caller, and it keeps going until one slow service melts the whole fleet. So the reframe is that a missing timeout is a way to spread contagion, not just a slow request. The real cost is thread exhaustion across the fleet. A deadline takes the unbounded hang and makes it a bounded, local failure that stays put.
a timeout that doesn't cancel and propagate just orphans the slow work
Firing a timeout feels like it stops the slow operation. It doesn't. A timeout that only walks away from the caller leaves the underlying work still running on the other side, and now that work is an orphan with nobody waiting for what it produces. It gets worse, because a retry that the timeout sets off can fire a duplicate, so now you've doubled the load instead of cancelling anything. The lesson here is that a deadline is only safe when the thing it bounds can actually be cancelled, so the timeout propagates and tears the work down, or when it's idempotent on retry, so a duplicate does no harm. Otherwise timing out just makes the overload worse.
a deadline that murders correct work at hour five
Deadlines feel like they only ever protect you, so it's easy to think a timeout could never be the thing that goes wrong. But put a wall-clock timeout on work that is legitimately long and still making progress, say a six-hour training run, and you kill correct work whose only crime was needing time. This is where the deadline rule stops. Deadlines are there to catch hangs, where nothing is happening at all, not slow work that's still moving along. Work that's genuinely getting somewhere needs a progress signal, a heartbeat, or a cancellation token rather than a blunt wall-clock cut-off.
VII. Bound what callers can create; release what you acquire
A missing limit isn't a missing feature, it's an unbounded liability
An absent cap looks like one fewer thing to build, the YAGNI virtue of not adding what you don't need yet. But leaving out a ceiling on caller-driven growth isn't putting off a nice-to-have. It's shipping a latent denial of service, or an out-of-memory, that goes off later in production. YAGNI is about features. A limit is a safety requirement, and you don't skip it just because dev traffic never got near it. That's how the tenet's tension between keeping things simple and keeping things safe gets resolved.
"Unbounded" is just a synonym for "fails later, mysteriously, in production"
"Unbounded" reads like a neutral technical word for something that can grow. The point is that growth with no ceiling isn't harmless. Anything without an explicit limit gets pushed past safe limits sooner or later, by real traffic or accumulated data or some hostile input, and when it goes it fails a long way from where the missing bound actually lives, which makes it a pain to trace back. Naming the real consequence, a future incident that's baffling precisely because the cause and the symptom are split apart in time and space, retrains you to read every unbounded list, loop, retry, buffer or allocation as a production failure waiting to happen rather than as code you can live with.
wait-or-reject
Wait-or-reject is the policy you give a bounded resource pool. A caller asking for a slot either waits a bounded amount of time for one to free up, or it gets rejected outright, usually with a 503, but it never blocks forever. It names the safe way a fixed-size pool falls over under pressure. Compare that with unbounded resource creation, where every new caller spawns another connection or thread, and a traffic spike turns into a death spiral that drains the machine. The discipline is to make scarcity an explicit, bounded thing the caller can deal with, rather than hiding it behind unlimited growth that fails catastrophically later on.
a flaky upstream becomes a self-inflicted DDoS
People assume a distributed denial of service comes from some external attacker. But uncapped retries against an unreliable dependency mean your own retry traffic piles onto a service that's already struggling, looking exactly like an attack, except you're the one who launched it on yourself. So without a cap you become the attacker, and the second the dependency wobbles your retries amplify that wobble into a flood. The fix is retry caps with exponential backoff and jitter, so your traffic backs off instead of surging when things go bad.
an orphan-in-waiting unless something releases it
Anything you acquire (a goroutine, a timer, a subprocess, a subscription, a file handle, a lock) will leak unless exactly one owner releases it on every exit path, the error paths included. Orphan is the word for a resource left with no owner to clean it up. In-waiting is there because the leak is latent and looks harmless right up until the release that never comes. In garbage-collected and scripting languages this still bites you, because the runtime reclaims memory but never external resources, so you reach for the language's scoping construct (with, defer, try/finally, RAII) where there is one and release by hand only where there isn't.
the runtime reclaims memory and nothing else
In a garbage-collected language it's easy to assume cleanup is taken care of for you. The collector only frees memory. It never releases subscriptions, timers, subprocesses, file handles, sockets, or locks. Those are external resources, and the runtime knows nothing about them, so they stay yours to release by hand. Believe the GC handles all of it and you'll silently leak every non-memory resource, and that leak is the thing you end up chasing in production when handles or connections run out long before memory does.
YAGNI
YAGNI stands for "You Aren't Gonna Need It", Kent Beck's Extreme Programming rule against building speculative flexibility for futures that may never turn up: write the simple thing now, not the configurable, generalised version you imagine you might want one day. The tenet scopes it carefully to features, not to limits or failure handling, and that scoping is the subtle bit. A missing feature is just simplicity you can bolt on later, but a missing limit or an absent timeout is a latent liability, so deciding not to add one isn't YAGNI. It's leaving the blast radius unbounded.
VIII. Tear down what you set up; subscribe for latency, reconcile for correctness
subscribe for latency, reconcile for correctness
Use events or subscriptions for fast notification, since they tell you about a change with low latency, but keep a periodic reconciliation sweep that re-reads the actual state to catch everything the events are bound to miss. This pairing compresses the edge-versus-level-triggered distinction. Events are sharp and cheap but lossy, because a dropped, reordered, or duplicated message leaves you silently wrong, whereas a convergent reconciler that compares desired against observed state heals that drift on its next pass. You get the speed of events for the common case and the correctness of polling as a safety net, and you don't have to pick one.
prefer the event over the timer, except where the event is silence
This reads like it contradicts itself, preferring events and then carving out an exception. But the two halves are about different failures. Normally you find out something finished from a real event, which is precise and arrives straight away. When the failure mode is the absence of any event at all, a crash, a hang, a silent disconnect, there's no edge to subscribe to, so only a timer or heartbeat can spot that nothing showed up. The rule is that polling is exactly the right tool when the truth you're watching for is silence.
level-triggered, convergent reconciler
A level-triggered reconciler is a loop that acts on the current observed state (the level) rather than on transitions (the edge), and convergent means it keeps driving the system back toward the desired state, so re-running it is always safe and gets there eventually. Contrast that with edge-triggered handlers, which fire once on a change event. If that one event is lost, reordered, or just never shows up, the handler never runs and the system stays wrong forever. A level-triggered loop re-examines reality on every pass instead, so a missed event just gets corrected next time round. That is what makes it the reliable backbone behind lossy event streams.
reference strength
Reference strength is whether a reference keeps its target alive (strong) or just points at it without preventing collection (weak). You pick this explicitly in manual-memory and ARC languages such as Rust, C++, and Swift. It has nothing to do with how "important" a reference is - it directly controls object lifetime. The strong-versus-weak decision is what sets the failure mode. A strong reference from the long-lived side wedges teardown and leaks. A weak one risks the target vanishing while you are still using it. So getting the strength right is how you dodge both the leak and the dangling watcher.
a strong claim from the long-lived side wedges teardown and leaks; a weak claim risks the watched thing being collected
This describes the reference-strength trade in manual-memory or ARC environments. A strong reference held by the long-lived side - a cache, a registry, an observer - keeps its target alive forever, so teardown wedges and you leak, often as a retain cycle where neither object can be freed. A weak reference avoids that, but it lets the target be garbage-collected or evicted out from under a still-active user, and now you have got the listener that silently stops firing or the cache entry that disappears mid-use. The fix is directional. Hold weak when the watcher outlives the watched, and strong-with-explicit-teardown when it does not, matching reference strength to whichever side is expected to die first.
retain cycles
A retain cycle is two objects holding strong references to each other, so each keeps the other alive and neither is ever freed, even once nothing else refers to them. It is the canonical leak in reference-counted systems, and it is the concrete failure that the whole notion of choosing reference strength exists to prevent. It is also what grounds the rule about holding a weak reference on the long-lived side - break one arm of the cycle, by making the parent-to-child or observer link weak, and the count can fall to zero so the memory gets reclaimed.
IX. Check-then-act on shared state is a race unless it's atomic or serialised
Check-then-act on shared state is a race unless it's atomic or serialised
Reading a condition and then acting on it - if the slot is free, take it; if the balance covers it, deduct it - looks like obviously correct sequential logic. But it is really two separate operations with a window between them. If any other actor can slip in during that window, the value you checked may already be stale by the time you act, so the check was only ever a guess. Three things close the gap. You can make check and act one indivisible step (atomicity, e.g. compare-and-swap), or let a single owner do both with no concurrency at all (serialisation), or design the operation so order does not matter in the first place (commutativity). The nasty part is that the window is invisible in code review and only turns devastating under real concurrent load in production.
TOCTOU
TOCTOU stands for time-of-check to time-of-use - the bug where the state you checked changes between the moment you checked it and the moment you rely on that check. It is the formal, well-studied name for the check-then-act race, drawn from the security and concurrency literature, where it famously enables exploits like swapping a file between an access check and the open that uses it. Naming it ties the manifesto's plain-language rule to a recognised flaw that already has a known body of defences, so you can go looking for the right primitive instead of reinventing it.
every one of these is two operations pretending to be one
Constructs like if not exists then create, or if balance is enough then debit, read as a single atomic intent sitting on one line. At runtime they are a read followed by a separate write, with an exploitable gap in between. Naming the disguise is the whole insight - the bug is believing the two-step is indivisible when the runtime makes no such promise, so two actors can interleave in that window. It only bites where interleaving can actually happen, which means concurrent requests, multiple workers, overlapping cron runs. The fixes are to make the step genuinely atomic, serialise it under one owner, or make it idempotent so the race does no harm.
the check is a guess
In a check-then-act sequence you read some state, decide based on it, and then act. If any other actor can change that state between the read and the act, the condition you verified may already be false by the time you rely on it, so the check told you nothing you can lean on. The phrase reframes such a check as worthless under concurrency - a value you read but cannot hold, whether with a lock, a transaction or an atomic compare-and-swap, is no better than a guess. The remedy is to make the decision and the action one atomic step, or else hold the state so it cannot move underneath you.
"it's single-threaded" is the assumption that turns out wrong most often at scale
Code that looks race-free because it appears to run alone is the most common false premise in production. The assumption quietly breaks the moment there are concurrent requests, or multiple worker processes, or several instances behind a load balancer, or overlapping cron runs you never planned for. The danger is specific. "Single-threaded" is the very belief that makes a race window seem impossible, so it is the assumption you stop guarding against right before it stops being true. Treat genuine single-threadedness as something you have to enforce and prove, not as a comfortable default you inherit.
compare-and-swap
Compare-and-swap is an atomic, hardware-supported operation that updates a value only if it still equals the version you previously read, and otherwise fails so you can retry with the new value. It is the primitive behind optimistic concurrency - rather than locking, you read, compute a new value, and try to swap it in on the condition that nothing has changed underneath you. What matters is that it collapses the racy check-then-act sequence into a single indivisible step, so the window in which another actor could interleave just is not there.
fenced lease
A fenced lease is a time-bounded lock (the lease) that also carries a monotonically increasing token (the fence), so any write tagged with a stale token gets rejected by the resource being protected. Martin Kleppmann describes the failure it fixes. A process acquires a lease, then pauses - a long GC pause, a scheduler stall - and the lease expires and is granted to someone else, and then the paused process wakes up still believing it holds the lock and tries to write. The fence number defeats this, because the resource has already seen a higher token and refuses the stale holder's write, which a plain expiring lease on its own cannot prevent.
expect to be wrong about holding it
Once you have acquired a lease you naturally assume you still hold it, and doubting that feels paranoid. But across a distributed boundary your belief can be stale - a network partition or an expiry may have revoked the lease without telling you. The check that you hold the lease and the act that relies on it can be separated by an arbitrary gap, so you guard the act with a fence token the resource validates rather than trusting your earlier acquisition. Design as though your held-lock belief may already be false at the moment you act on it.
claim work by atomic rename, not by checking a \"processing\" flag
To claim a unit of work safely, do an atomic rename - move the job file into a per-worker or in-progress name - and let the filesystem guarantee that only one renamer wins, rather than reading a "processing" flag and then setting it as two separate steps. The flag approach is a classic check-then-act race, where two overlapping cron runs can both read the flag as unset and then both go ahead and claim the same job. A rename is indivisible at the OS level, so exactly one contender succeeds and the rest get a clear failure back. That makes it the simple, lock-free way to get single ownership of a task.
X. Make operations idempotent so "do it again" is always safe
the only safe operation is one that's harmless to repeat
This sounds absolutist, and plenty of one-shot operations do seem perfectly safe. But once you are dealing with retries, redelivery and at-least-once messaging, any operation whose effect differs between running once and running twice is going to get run twice eventually, and corrupt something. So safety here means idempotence: you structure the operation so a repeat is a no-op or just produces the same result. Re-delivery is going to happen at scale, and harmless-to-repeat is the one property that survives the retries you can't prevent.
every retry becomes a corruption
Retries are the usual tool for surviving transient failures. They are only safe if the operation is idempotent, meaning applying it twice has the same effect as applying it once. When that doesn't hold - a retry double-charges, or double-increments, or double-sends - the very mechanism you lean on for reliability turns into the thing that wrecks your state. The phrase spells out the exact condition for the harm: if once differs from twice, then under the retries that networks and clients are going to perform anyway, corruption isn't a risk, it's a certainty. That is the reason to make operations idempotent, rather than to try to avoid retries.
make creates upserts
Turn an insert into an insert-or-update keyed by a stable, caller-supplied id - an upsert - so that creating the same thing twice converges to a single row instead of producing a duplicate or throwing a uniqueness error. This is the standard way to make creation idempotent, and it matters because retries are going to happen: a network blip can make a client resend a create it isn't sure landed. With an upsert keyed by id, that retry safely reconverges to one record. A plain insert would either error out or quietly create two.
exactly-once where it counts
This sounds like a contradiction, or like a promise of true exactly-once delivery, which you can't have. What it actually means is that you can't get genuine exactly-once delivery from the transport, but consumer-side deduplication by message id makes duplicate deliveries have no observable effect, which is the thing you wanted in the first place. The qualifier where it counts points at the places that matter, the ones with side effects, and that is where you spend the effort to dedupe. What matters is the effect being applied once, not the delivery count being one.
exactly-once is faked
Messaging systems advertise exactly-once, so it sounds like a real guarantee you can buy. But true exactly-once delivery is impossible. The lost acknowledgement of the Two Generals problem means a sender can never know whether a message was received, so it has to either risk losing it or risk sending it twice. What gets marketed as exactly-once is at-least-once delivery plus idempotent deduplication, simulating the effect at the edges. Any claim of real exactly-once is faking it at the application layer, and you want to know where that faking happens.
XI. Separate the irreversible decision from its effect
Separate the irreversible decision from its effect
Deciding to delete and actually deleting feel like one natural action, so splitting them looks like pointless indirection. But the split is what makes correctness testable. You make the decision of what or whether to do something a pure function that returns a plan, and you make the doing a thin wrapper that just executes that plan. A delete() that decides as it deletes can't be tested without actually deleting, whereas a pure planner can be hammered with every edge case while the executor stays trivially correct, because it carries no logic of its own.
functional core, imperative shell
An architectural shape where all the real decision-making lives in a pure core that has no side effects, takes data in and returns data out, and so is trivial to test in isolation. That core gets wrapped by a thin imperative shell whose only job is to do the actual effects - write to the database, send the request, print the file. It is Gary Bernhardt's name for the structural form of separating an irreversible decision from its effect: the part that decides what to do is pure and can be tested exhaustively, and the part that does it is small enough to eyeball. Keeping logic out of the shell means almost all your behaviour is covered by fast deterministic tests, and the risky effectful code stays too dumb to hide bugs.
dry-run
Running the full decision path to produce and show the plan of what would happen, while doing none of the real effects. This is only possible once you have split the decision from its effect: the pure core computes the intended changes, and the shell, instead of applying them, prints them. It lets a human inspect exactly what is about to be created, deleted or migrated before anything irreversible happens, which is why tools like terraform plan or a --dry-run flag are the standard safety net for dangerous operations.
four-eyes
A control that needs two people, hence four eyes, to approve an irreversible or high-stakes action before it goes ahead. It names the human review that the decision-and-effect seam makes possible: because the decision is produced and shown before the effect fires, a second person can check it in between. Tenet XXV takes this from a mere capability and turns it into a process you actually run for the riskiest changes, so the safeguard isn't just available, it's required.
The catastrophic verbs - delete, kill, charge, send, overwrite, launch - are dangerous mostly because the reasoning is fused right into them
Irreversible operations are dangerous not because the words are scary but because of a structural defect: the decision of what to act on is welded into the side-effecting code. A delete that decides what to delete as it deletes can never be tested without actually deleting something, so you end up under-testing exactly the code that most needs proof. Fused names that inseparability of decision and effect. The fix is to split them, so a pure function like whichRowsToPurge can be hammered with thousands of cases and zero side effects, while the tiny wrapper that takes those ids and runs the delete is small enough to read in one breath. The same seam is what makes dry-run, four-eyes approval and undo possible.
A delete() that works out what to delete while it's deleting can't be tested without something actually getting deleted
When the logic that decides what should be destroyed is fused to the act of destroying it, the only way to exercise the decision is to trigger the destruction, so the decision goes effectively untested. You wind up with no proof for exactly the code whose mistakes are least recoverable. Splitting the decision from the effect - compute the set to delete, then delete it - creates a seam, and that seam is what makes dry-run previews, four-eyes review of the proposed deletions, and undo possible. The principle generalises to any irreversible action: separate the choice from the consequence so the choice can be inspected before it fires.
how you erase the wrong directory at 2am
Fusing the selection and the destructive action into a single command, like a find with delete that chooses and deletes in one breath, is the structural setup for catastrophic, irreversible operator error. The 2am shorthand stands for a tired human with no safety net, no dry-run, no review, no undo. Naming it this way makes the point that the failure is structural and not a personal lapse. The cure is to separate planning from applying - terraform plan, then apply - so the dangerous choice can be inspected before it takes effect, not to tell people to be more careful when they are exhausted.
The plan goes stale, a TOCTOU window (IX)
Splitting the decision from the effect was supposed to take a hazard away, so it comes as a surprise that the split can put one back. Here is why. The plan gets computed against a snapshot of the world, and if the world moves on before you apply it, then running the plan blind is a time-of-check-to-time-of-use race, the same one as in tenet IX. That gap you opened up to get testability is also a gap where the assumptions can rot. So pin the plan to a version or a snapshot id, and when you go to apply it, check again that the world still looks the way it did.
XII. Finish your obligations before you exit
The OS reclaims memory, never meaning
When a process exits the operating system frees its RAM for you, and its file handles, and its sockets. What it does nothing about is the semantic obligations the process had picked up along the way: writes buffered but never flushed to disk, messages consumed but never acknowledged, requests dropped halfway through a handshake. Those are promises made to other parties downstream, and the only thing that can keep them is your own shutdown code, because the runtime has no clue they exist. The line is drawing a hard wall between the resources the system tidies up on its own and the meaning it just cannot, and that is the whole reason a clean exit can still quietly lose work that somebody else was relying on.
a half-written CSV never gets mistaken for a finished one
A partial output file wearing the final name is dangerous for the worst possible reason - nothing about it looks off. A reader sees the path it expected and treats the truncated data as the real thing, and from there every step downstream gets quietly corrupted. So write to a temporary path, and rename it across to the final name only once the write succeeds, atomically. Now the destination name shows up only after all the data is there. That cuts what a reader can possibly see down to two states, complete or absent, and it gets rid of the third lying state where a file that looks finished is really just a fragment a crash left behind.
temp-write-then-rename
The crash-safe way to produce a file. You write the whole output to a temporary path first, and then you do one atomic rename to the final name, once the write has fully gone through. Rename on a local filesystem is atomic, so a crash at any point before it just leaves an orphan temp file lying around, never a half-written file wearing the real name, and a reader gets either the complete result or nothing. This is the same trick maildir uses for crash safety. It is the floor for anything that writes out a file, right down to a throwaway one-shot script.
ack-before-persist silently eats data on restart
Acknowledge a message, or commit a stream offset, before you have actually written the record down durably, and a crash in that small window between the ack and the write loses the record. The source system meanwhile thinks the record was consumed fine, so it never resends it. The ordering has to be strict here: persist the record, then acknowledge it. Doing it the other way round, ack and then save, feels like the same thing and feels harmless, but it turns at-least-once delivery into silent data loss, and the loss only shows up on restart, with no error to point at, because every single component reckons it did its part.
a WAL
A write-ahead log. Before you apply a change to your real state, you first record what you intend to do, durably, to an append-only log, so that if the process gets killed dead the change can be replayed and finished on restart. It is one of the core things that makes an abrupt exit survivable by design rather than by luck, because the durable record of what you meant to do outlives the in-memory work that did not. Put a WAL together with idempotent writes and recovery is just replaying the log, and a crash partway through an operation costs you nothing you had already promised to write down durably.
crash-only
A design discipline where the only way to stop a component is to crash it, which means recovery has to be the normal startup path instead of some special case. There is no separate graceful-shutdown routine, so there is no rarely-run cleanup code sitting around being easy to get wrong, and every restart puts the same recovery logic through its paces that you lean on after a real failure. The idea comes from Candea and Fox: if crashing is always survivable and recovery is the everyday path, you never build up a fragile clean-shutdown branch that only ever runs when it is already too late to test it properly.
XIII. Failure modes must be visible and impossible to swallow
the sin is the silent swallow, not the keyword
The language wars set this up as exceptions versus Result versus error codes, as if the mechanism were the part that mattered. But the mechanism is just idiom. The one actual sin is letting a failure disappear quietly, nobody told, nothing logged. That pulls the invariant - failure has to stay visible - apart from the neutral mechanism that carries it around. A typed exception caught and handled at a boundary is proper, first-class error handling, not some fallback, and an empty catch block is the failure no matter which keyword got you there.
A swallowed error is a wrong state that has learned to hide
Throwing an error away in an empty catch block does not get rid of the failure. It hides a state that is now corrupt, so the thing comes back later, miles from where it started and far harder to track down. Talking about the swallowed error as something that has learned to hide is the point of those blocks being time-bombs rather than tidy little cleanups: the wrong state is still there, only now there is nothing at all pointing at it. What the tenet is after is that failure stays visible and stays impossible to ignore, whether you carry it in a Result, or a (value, err) pair, or typed exceptions caught at a boundary you chose on purpose. Just never dropped on the quiet.
the happy path a lie
When failures tunnel up out of sight as exceptions, through frame after frame, the reader looking at any one frame cannot see what might blow up underneath it, so code that seems to succeed is actively lying about what is going on below. Calling the happy path a lie casts swallowed or hidden errors as deception, not just untidiness. You cannot trust the success path when any frame down below it might have failed without a word. So make failure visible and impossible to ignore at every frame, and then apparent success actually means success.
EAFP
Short for Easier to Ask Forgiveness than Permission. It is Python's idiomatic style of just trying the operation and catching the exception if it goes wrong, rather than checking every precondition up front, which is the opposite style, Look Before You Leap, or LBYL. The manifesto brings it up to make one precise point: in languages where exceptions are the native idiom, a typed exception caught at a boundary you picked deliberately is a first-class way to make failure visible and impossible to ignore, not some second-rate fallback to errors-as-values. The discipline that counts is catching at a real boundary with a specific type. Never a bare except that swallows the lot.
set -euo pipefail
The Bash incantation that makes a script fail fast. The -e exits on any command that comes back non-zero, the -u treats using an unset variable as an error, and -o pipefail makes a pipeline fail if any stage of it fails rather than only looking at the last command's status. This is the fail-fast mechanism for glue and scripting, where the dangerous default runs the other way: a failed step gets ignored quietly while the script ploughs on and wrecks whatever the next step touches. Stick this at the top of a script and a silent partial failure becomes a loud halt right where it happened.
Choose by blast radius: abort where a wrong result silently propagates, degrade where staying up with less is the lesser harm
This settles the real conflict between failing fast (tenet XIII) and degrading gracefully (tenet XIX) by going to blast radius, rather than to some blanket preference. Where carrying on with bad data would quietly corrupt downstream state, or push out a wrong answer that other people trust, you abort, because a loud stop is cheaper than corruption nobody notices. Where staying partly available really is the lesser harm, meaning you can serve stale or cut-down results without misleading anybody, you degrade rather than go dark. None of this is a vague "it depends". It is the same control-and-blast-radius tie-breaker that runs through the rest of the manifesto, here applied to the abort-versus-degrade call.
XIV. One source of truth; derive the rest
One source of truth; derive the rest
Every fact in the system has exactly one owner that's allowed to say what it is, and every other place that fact shows up is a computed or subscribed view of that owner. None of those other places is a second master you can write to directly. It sounds like a generic DRY slogan but the point is sharper than that. Store the same fact in two writable places and you've really got two facts, and sooner or later they'll disagree, and then the system is holding two contradictory beliefs and there's no way to tell which one is right. If you derive everything downstream instead - computing it, caching it, subscribing to it - then they can't drift apart by construction. It isn't just discouraged. It can't happen.
Declared divergence is fine; undeclared is the bug
Single source of truth doesn't mean you can never copy data. It means every copy has to be governed. A copy that has a named owner, a defined way to invalidate it and an explicit staleness budget - a cache, a read replica, a denormalised view - is fine, it's normal engineering, because how far it can drift from the source is declared and bounded. The actual defect is the unmanaged copy, the one with no owner and no reconciliation story, because nobody knows when it's gone stale or whose job it is to fix it. So the rule takes "never duplicate" and turns it into something more useful: every duplicate has to be declared and reconciled on purpose.
staleness budget
An explicit, declared limit on how out of date a derived copy - a cache, a replica, a materialised view - is allowed to be. Say no more than thirty seconds behind the source. It takes the vague worry that the cache might be stale and turns it into a named contract with a number on it, so everyone knows the most it can be off by and can reason about that. The value is the line it draws. On one side is divergence you engineered and accepted, inside the budget. On the other is an actual bug, where the copy has drifted further than you promised.
a parallel kingdom of truth you sync by hand and pray
This is a second writable copy of authoritative data - a hand-maintained client cache, say - that competes with the real source and gets reconciled by hand with no guarantee it stays correct. Kingdom of truth mocks the idea of a rival authority that shouldn't be there next to the real one, and sync by hand and pray admits the reconciliation has no actual mechanism behind it, so the thing drifts. The fix is that derived data, a React Query cache or a store, has to be an explicitly derived view with a named owner and an invalidation path, not some parallel source you keep lined up by remembering to.
XV. Name the boundary, version the contract: strict in, tolerant of the unknown
Name the boundary, version the contract: strict in, tolerant of the unknown
Strict and tolerant sound like they contradict each other in one instruction, but they're pointed at different things. You reject inputs that break the invariants you actually rely on - you're strict where correctness depends on the constraint holding. But fields you simply don't recognise, maybe from a newer or older peer, you ignore rather than throwing out the whole message. Being strict about the invariants and tolerant of the unknown additions is what lets a contract change over time without breaking consumers that can't upgrade at the same moment. It takes Postel's robustness principle and makes it precise, instead of leaving it as a vague be-liberal slogan.
tolerant readers
Consumers written to ignore fields they don't recognise instead of rejecting or crashing on them, so a producer can add new fields without breaking anyone downstream. It's Martin Fowler's term for the integration pattern that lets a schema grow additively: producers extend the message, old readers quietly skip the extras, and nobody has to upgrade at the same time as everyone else. That's the thing that makes it practical to deploy services that share a wire format independently, instead of needing a coordinated flag day.
A boundary you didn't design is one your bugs designed for you
This reads like a general remark about boundaries existing, but it's sharper than that. If you don't deliberately define an interface, an implicit and undocumented one builds up anyway, out of accidental coupling and incidental behaviour that callers end up relying on. This is Hyrum's Law: with enough consumers, every observable behaviour becomes a contract whether you meant it to or not. And once that accidental boundary sets, you can't safely change the implementation behind it any more. So the choice is to design the boundary on purpose, or let your bugs and accidents design it for you.
The boundary parser is now your most security-critical, most-tested code
A parser feels like plumbing. It's the boring code that turns bytes into objects, not where you'd expect the risk to be. But the code that accepts and interprets input at a boundary is exactly where hostile input lands first. A bug in it is a bug in everything downstream that trusted what it produced, because every later stage just assumes the parser already checked the data. The boundary parser is the one chokepoint every attack has to go through, and that's why it earns the most scrutiny and the heaviest test coverage, rather than getting treated as low-stakes glue.
a consumer on old code is not a bug, it's Tuesday
Clients running old versions of your contract are a routine, expected, permanent fact of life, not some exceptional fault you go hunting down and fixing. "It's Tuesday" is idiom for completely ordinary and unremarkable. Mobile apps you can't force-upgrade, events already sitting in a queue, third-party integrations, cached SDKs - all of them guarantee that some caller is always several releases behind. So backward compatibility is the baseline you build the server around, not a special case. An old consumer hitting your API is normal traffic. Plan for it by design instead of treating it like a misconfiguration.
\"everyone updates\" is a wish, not a deployment strategy
Telling users to upgrade sounds like a plan, but you can't actually make clients move to the new version on your schedule - mobile apps especially, but also queued events and embedded integrations - so leaning on universal upgrade is wishful thinking, not engineering. Wish versus strategy is the contrast that punctures the idea that you can deprecate an old contract just by shipping a new one. A real strategy accepts that your server has to keep serving consumers several releases back, and it designs the contract so old and new clients both work at once.
XVI. Least privilege, by construction
Least privilege, by construction
Give each component the narrowest authority it needs, and do it structurally - a read-only role, a scoped token - so that a component that gets compromised or just has a bug is held back by authority it was never handed, not by trust that it'll behave. The phrase that's doing the work is by construction. Least privilege is a familiar idea, but the insight is that the containment should come from the capability being absent, not from hoping the code stays well behaved. It's Saltzer and Schroeder's principle made concrete, and it's the runtime sibling of making over-reach unrepresentable in the type system. Cheaper and more reliable than trusting every caller.
containment is cheaper than trust
Over the life of the system, limiting what a component can touch costs less than relying on that component, and every future caller of it, to wield broad power correctly. It reads like a generic security platitude, but it's a specific design claim: least privilege is the runtime sibling of making illegal states unrepresentable. Instead of granting a capability and trusting it'll only ever get used well, you design the capability out, so the misuse just isn't possible. Trust has to be earned again on every change, by every caller. Containment you pay for once, in the structure, and then it holds.
the god-object
An object carrying far more data and authority than any single caller actually needs, so passing it around hands out sweeping power by default. It's the concrete counterexample to least privilege. When a function takes the whole god-object it can touch anything on it, and you can't tell from the signature what it really uses. The fix the tenet points at is the opposite of that: a function that takes only the two fields it touches, which both shows and limits how far it can reach.
Broad * IAM is the difference between an incident and a catastrophe
An access policy that grants all actions on all resources through wildcards is easy to write, but here is what it costs you: a single compromised component (a leaked key, an injected dependency, a confused-deputy bug) inherits authority over everything that policy can reach. The phrase puts a number on what over-broad permissions are worth in blast-radius terms. Scoped permissions turn a breach into a contained incident. Wildcard permissions take that same breach and spread it across your whole account. It names the canonical over-privilege anti-pattern so that a * in a policy reads as a future blast radius and not as some harmless shortcut.
XVII. Measure, then make the common case fast, but bound the unbounded without a profiler
Performance is a falsifiable property of the running system, not a vibe
Performance claims have to be measured against the actual running system under realistic conditions. You don't get to assert them from intuition or read them off the code by eye. The word "falsifiable" is doing the real work here: a genuine performance statement is one you could prove wrong with a profiler, and that is why you profile before you trade clarity for speed, and why you let measurement kill your guesses, because intuition is usually wrong about where the time actually goes, especially on cold paths. There is one carve-out, and it is unbounded algorithmic complexity driven by attacker-controlled or caller-controlled input. That you bound up front instead of waiting for a profiler, because the failure is structural and not a matter of constant factors.
bound the unbounded without a profiler
The tenet preaches measure first, so fixing performance without measuring sounds like a contradiction. But super-linear complexity in caller-controlled or attacker-controlled input is a correctness and denial-of-service bug, and you fix it on sight with no profiling required. Measure first governs which legible code you're allowed to make clever in pursuit of speed. It doesn't give you licence to ship algorithmic blowups that small dev datasets happened to hide. An accidental O(n squared) on untrusted input is a bug whether or not a profiler has flagged it yet.
\"dev data was fine\" is exactly how it ships
Code that runs acceptably on the small datasets developers test with passes review precisely because it looks fine, and then it ships and melts at production scale. "Dev data was fine" feels like reassuring evidence, but it's the trap and not the proof. An O(n squared) routine is instant on a hundred rows and catastrophic on a million, and the smallness of dev data is the very thing that hides the problem until it's sitting in front of real traffic. So attacker-controlled or caller-controlled algorithmic complexity cannot wait for a profiler to catch it later. The cheap-looking local result is the thing that lulls you into shipping the cliff.
the N+1 query
The pattern where you run one query to fetch a list of N rows and then, usually inside an innocent-looking loop, run one more query per row to load each row's related data, which gives you N+1 round-trips to the database where a single join or batched query would have done the job. It's named because the shape is recognisable enough to fix on sight without reaching for a profiler: a loop that lazily loads an association turns into hundreds of separate round-trips under real data. The cure is to fetch the related data in one go, by joining or batching, before the loop runs.
the chatty RPC
An interface that makes many small, fine-grained remote calls where one batched call would have done, paying the per-call network latency over and over again. Chatty here is a performance term of art for excessive round-trips, the distributed sibling of the N+1 query. It is not, as a newcomer might guess, a call that simply logs a lot or is verbose. Because each remote call carries fixed latency no matter the payload size, collapsing many chatty calls into one coarse-grained call is often the single biggest latency win on offer.
throughput is money
Unbounded fan-out, indefinitely retained data and always-on compute aren't merely sources of slowness. They are direct, recurring financial costs in the form of egress, storage, instance hours and per-request charges. The phrase reframes performance failures as cost failures, and that widens the reason to bound things past latency: even when a slow path is nowhere near a user's critical journey, its unbounded resource use still shows up on the bill. Thinking of throughput as money makes the case for sampling, capping and retiring resources that no latency budget alone would ever justify.
XVIII. Make it observable, or you are guessing
Make it observable, or you are guessing
Without signals that actually prove your invariants held in production, any statement you make about correctness is a guess and not a fact, because partial failures frequently leave no exception and no stack trace, just a quietly wrong outcome. Observability is the verification layer that makes every other tenet checkable in the running system rather than merely intended back at review time. You instrument the specific properties you claim are true so that you can see whether they actually are. The phrase goes past the generic "add logging" advice by tying the telemetry to the invariants it is meant to confirm.
You will not debug a partial failure from a stack trace, because often there isn't one
The stack trace is the reflexive debugging artefact, so expecting one feels reasonable. But partial failures in distributed systems turn up as latency cliffs, rising error rates, or creeping queue depth, with no exception thrown anywhere for you to catch. The interesting production failures are statistical and silent rather than a clean crash at a single line. This is why you have to instrument rates and trends and watch the distribution, instead of sitting there waiting for a throw that is never going to come.
The model didn't throw anything, it just quietly got worse as the world moved on
An ML model can raise no exception at all while its predictions steadily degrade, because the real-world input distribution drifts away from the data it was trained on. The world moved names this data and concept drift: nothing in the code broke, but the assumptions underneath it expired. There is no stack trace for silent degradation, so you can't rely on errors to tell you, and you have to track drift and freshness metrics instead, comparing records in versus records out, prediction distributions, and input null rates, to catch a failure that never throws.
a silent fallback nobody noticed for a week
A degraded mode kicked in and kept serving wrong-but-non-erroring results, which masked the fact that the primary path had been broken for days. This shows that even a successful, graceful degradation needs observability, because without it the fallback hides a real problem instead of surfacing it. You won't see this in a stack trace, only in metrics like a rising error rate that never came, source-versus-cache drift, or a fallback counter climbing. So the tenet's point is that degrading safely and reporting that you degraded are two separate requirements, and skipping the second one turns a safety mechanism into a concealment.
this is not "log everything", because noise is its own kind of failure
Observability is not the same thing as maximal logging. Excessive telemetry is a failure mode in its own right: it buries the few signals that matter under sheer volume, it costs real money to emit and store, and it risks leaking sensitive data into logs. The discipline is to emit the small set of signals that prove your invariants and nothing more, which means observability is itself bounded against the bound-everything tenet (VII). Telemetry is an unbounded resource you have to sample and cap like any other. More logging is not always better. The right logging is.
XIX. Degrade in tiers; contain the blast radius
Degrade in tiers; contain the blast radius
When a dependency dies, deliberately shed one feature rather than letting the whole system fall over, and isolate failures so one tenant's bad day doesn't become everyone's. The two clauses name the two halves of fault tolerance. Degrade in tiers is the temporal half, where you choose in advance what to lose first and lose it gracefully. Contain the blast radius is the spatial half, where you bound how far a failure is allowed to spread. Although degrade sounds like something you'd want to avoid, deliberate degradation is exactly what keeps an incident from turning into a full outage.
a circuit breaker that fails fast to give the downstream some room to heal
Failing fast on purpose sounds like giving up rather than helping. But when a downstream is struggling, rejecting requests quickly stops you piling more load onto it, so it gets a chance to actually recover instead of being hammered flat by traffic it cannot serve. If you keep retrying against a sick dependency you stop it recovering at all, which is why failing fast is the cooperative move here. The reason it's safe is that the surrounding operations are idempotent and deadlined, so anything you shed can be retried later without harm once the breaker closes again.
bulkheads
Isolated resource pools, one per dependency or per tenant. So a separate thread pool or connection pool for each downstream service, which means one compartment filling up can't drain the resources the rest of the system needs. The name comes from naval architecture, where watertight bulkheads keep a flood in one compartment from sinking the whole ship. In software a slow or failing downstream exhausts only its own pool and stalls only its own callers, and everything that doesn't depend on it carries on serving as normal.
thundering herd
This is what happens when a lot of clients that failed together all retry at the same instant, hammering a recovering service all at once and knocking it back over just as it's trying to come up. There's no actual stampede involved, it's a synchronisation problem. A shared failure lines up everyone's retry clocks, so the moment of recovery turns into one coordinated spike. Naming the thing is what justifies the cure, which is jitter, because randomising each client's backoff pulls the retries out of sync and spreads the load so the service can recover for real.
jitter
Randomness added on purpose to retry and backoff delays, so that clients which failed together don't all retry at the same moment. It's got nothing to do with nervous shaking or signal noise here, it's an engineered spread. Jitter is the specific antidote to the thundering herd. By scattering retries across a window instead of firing them all in lockstep, it flattens the recovery-time load spike that would otherwise overwhelm the service all over again.
cell isolation
Splitting a system into independent cells, each one serving a subset of tenants or traffic with its own resources, so that a failure stays inside the single cell where it happened and can't spread out across the whole system. This is AWS's fault-isolation architecture, and the term comes from that lineage rather than from biology or prisons. What you get out of it is a bounded blast radius. A bad deploy or a poison input takes down one cell's fraction of customers rather than the entire fleet.
white-screening
When an unhandled error in a single frontend component crashes the whole application and the user ends up staring at a blank white page instead of a contained fallback. Despite how it reads, it's not a styling thing about a white background, it's the catastrophic failure mode of a UI that has no error boundaries. The fix is an error boundary that catches the exception and degrades just the one widget, so a broken component shows a small fallback while the rest of the page keeps working.
an untested degraded path is just a second bug waiting for the worst moment to fire
A fallback path feels like extra safety, obviously better than having none at all. But a degraded path you never exercise is just unverified code, and it'll most likely fail at exactly the moment the primary path already has, which is the worst possible time. Fallbacks are real complexity, not free insurance. Only the tiers you actually rehearse, through drills or fault injection, give you any safety. The rest are latent bugs you haven't triggered yet, and an incident is a bad time to find out your safety net was never wired up.
XX. Optimise for reversibility, deletion, and change
build a seam only when you can name the second concrete thing that will go through it
Building abstractions and seams up front looks like sensible forward-looking design, but you shouldn't add an abstraction layer for a single hypothetical future. You build it only once you can point at a second real, concrete use that has an owner and a date. One use needs no abstraction, and two of them reveal where the variation actually falls. Speculative flexibility is about the most expensive clutter there is. A guessed-at seam is usually wrong about where the variation lands, so you pay for the indirection and get none of the benefit.
they're bets you've forbidden yourself from losing gracefully
A big-bang rewrite or a one-shot irreversible migration feels bold and decisive, but it takes away any route of retreat, so if the bet turns out wrong you fail catastrophically with no way back. The reframe recasts those bold moves as reckless wagers that you've set up so you can't lose gracefully. Reversibility - incremental rollout, dual-running, the ability to roll back - lets you find out you were wrong cheaply and undo it, which turns a catastrophic bet into something you can survive and experiment with.
expand-then-contract migrations
A schema change done as a sequence of small reversible steps instead of one risky cut-over. First you expand by adding the new shape alongside the old, then you backfill and dual-write so both are populated, then you switch reads to the new shape, and only at the end do you contract by removing the old. The defining property is that every intermediate state is fully runnable and rollback-able, so you can stop or reverse at any point. It's the safe alternative to a one-shot drop-and-rename, which is a cliff with no way back if anything goes wrong mid-deploy.
the drop-and-rename in one transaction is a cliff
Atomically replacing a schema object in a single irreversible transaction leaves no safe intermediate state to roll back to. Once it commits there's no gradual retreat. Cliff marks that point of no return, as against the expand then backfill then dual-write then switch reads then drop sequence, where every intermediate state stays runnable and reversible. The principle is that schema changes ought to keep a safe path backwards at each step, so a problem you find mid-migration can be undone rather than forcing you over the edge in one move.
zombie flags
These are feature flags that outlived the rollout they were created for and never got cleaned up. They linger as permanent dead branches that nobody dares delete, because no one is sure what still depends on them. They're the cost of building seams without giving them an owner and an expiry. The remedy is concrete. Every flag and scaffold needs a named owner and a sunset date, or they pile up until you're drowning in them, and each one adds a dormant branch that can spring back in an incident exactly the way tenet XXI warns.
an archaeology dig for all its tendrils
Removing a tightly coupled component forces you to dig out every dependency it sent threading through the codebase, tracking down scattered call sites, shared state and implicit assumptions before you can pull it out. The phrase is the negative case that makes the argument for build-time reversibility. A piece that sits behind a narrow interface, in one folder or behind one flag, can be deleted in a single move because nothing reached past the boundary. When coupling is loose, deletion is surgery. When it's tight, deletion turns into archaeology, and the cost of removal tracks how many tendrils you let grow.
reversibility collides with a real duty to forget
Keeping everything for recoverability sounds like pure upside, the more undo the better. But soft-deleting and retaining data for reversibility runs straight into legal and security obligations to actually destroy certain data, such as personal data under privacy law. Data you keep is breach surface and legal liability, not just a safety net. The resolution works per data class. Some data needs a true hard-delete path and a retention policy that's encoded and enforced, so reversibility is bounded by the duty to forget rather than overriding it.
a real duty to forget
A real legal and ethical duty to actually destroy certain data, coming out of deletion rights and retention limits such as GDPR's right to erasure, and not some vague feeling about letting go. It works as the deliberate counterweight to reversibility. The same instinct that tells you to keep everything so you can roll back runs straight into the fact that data you hang on to is breach surface and legal liability. What the tenet is getting at is that default-keep-forever is the wrong default, and retention has to be decided per data class with a real hard-delete path, a TTL or a droppable partition for whatever you are obliged to destroy.
XXI. Simplicity is the budget that funds everything else
Software is the only material where having more of it makes the rest heavier
Add more of any physical material and you just get a bigger pile, but every line of code you add makes the code you already have harder to follow and harder to change. The weight here is cognitive, not physical, and it grows faster than the line count does because of new coupling, more state to keep track of, and more failure surface you have to reason about. The pun is that software, the stuff that was meant to be soft and easy to reshape, hardens as it piles up. This is the grounding case for treating subtraction as actual work, and it echoes Lehman's law that a system's complexity rises unless someone deliberately brings it back down.
Subtraction is real progress
Progress instinctively means adding features and code, so deleting can feel like going backwards. But removing code, state, and branches is genuine forward progress and not just tidying up, because it kills off whole classes of bugs that can no longer happen at all. A state that does not exist can't be corrupted, and a branch that is gone can't come back during an incident. The line pushes against the deep bias that output equals lines added. Removing the case beats handling it, and the deletion is often the single most valuable change in the whole diff.
The state you don't have can't be wrong
Take out a state, a case or a configuration option entirely and you remove the whole class of bugs it could ever have produced, which is more reliable than handling that state carefully. A stateless handler can't hold stale state, a config key that does not exist can't be misconfigured, and a field you never store can't fall out of sync with anything. It reads like a glib tautology, but it is really a design heuristic. The most dependable way to handle a failure mode is to arrange things so it does not exist in the first place, so reach for fewer states before you reach for more guards around the states you kept.
The dead branch you leave behind is the one that springs back on you in an incident.
Code that looks unreachable feels harmless, just inert lines sitting there unused. But the branches you fail to delete turn out to be exactly the ones that unexpectedly come back to life and cause the next outage, whether through a flag flipping, a config change, or some edge case you forgot the branch was there to handle. "Springs back" personifies dead code as dormant rather than gone, ready to wake up at the worst possible moment. Treating deletion as failure prevention and not just tidiness follows from the tenet's preference for fewer states over more guards, because a branch that no longer exists cannot fire.
a config DSL that reinvents a programming language badly
A configuration format that started out declarative but grew conditionals, variables, loops and interpolation until it is effectively a programming language, except an ad-hoc one nobody designed on purpose, with no debugger, no types and no tests. It can read as just a powerful config system, but it is really an instance of Greenspun's tenth rule applied to config. Pushing logic into configuration to dodge writing code does not remove the complexity. It just relocates it into a worse language. The tenet wants the opposite: a few small obvious services in a real language rather than one flexible engine driven by such a DSL.
pay for the tier when the problem bills you, not before
Adopt the next, heavier level of tooling (a SQL query, then a Spark job, then a bespoke framework) only when the data size or the load actually forces it on you, never ahead of time. This is YAGNI applied to architectural heft. Complexity should be a response to a real bill the problem hands you, not a pre-payment against some future that may never turn up. The pricing-tier image makes the trade concrete, because paying for capacity before you need it is waste, and the cheaper tier keeps the system simpler until reality genuinely demands more.
XXII. Name to reveal, not to label
Name to reveal, not to label
Naming looks like just naming, with no obvious gap between revealing and labelling. But a name ought to encode the one load-bearing fact a reader would otherwise get wrong, like the unit, the ordering guarantee, or a hidden side effect, rather than slapping a generic tag on the thing. A label like data or handle tells the reader nothing, whereas a revealing name like timeoutMillis or sortedByName lets them skip reading the body altogether. The stakes are lopsided here, because a misleading name installs a false mental model that is worse than having no name at all.
a misleading name is worse than none
Any name seems better than an anonymous or vague one. But a wrong name actively installs a false mental model that the reader trusts and then debugs against for an hour before they work out the code does something else entirely, which costs far more than the absence of a name ever would. Names are the most-read part of any codebase, so disinformation in them is high-leverage harm that spreads to everyone who reads the code. That is why precision matters more than coverage, because an honest absence is cheaper than a confident lie.
the load-bearing fact
The single piece of information that the code's correctness actually depends on the reader knowing, such as the unit a number is in, the order in which calls have to be made, or the fact that a function has a side effect. The construction metaphor is exact. It is the one detail holding the structure up, the thing a reader gets wrong without it. It tells you what to put in a name. Not everything, but specifically the fact the type cannot already carry, so the name earns its length by encoding the one thing that would otherwise be invisible.
A name that encodes an invariant is, in effect, a duplicate of that invariant
When you bake an invariant into an identifier, like chargeOnceIdempotent, the name is now a second copy of a fact that lives somewhere else in the code, and like any duplicate it can drift out of sync. The day idempotency gets dropped, the name still asserts it, so the name lies, which is exactly the single-source-of-truth drift that tenet XIV warns about. The lesson is to let the type carry the invariant wherever a type can, and keep name-encoding for facts no type captures, like units, ordering, or the presence of a side effect, while accepting that those names have to be renamed everywhere the moment the fact changes.
getUserByIdWithRetryAndCacheFromPrimaryReplica
This is a deliberately absurd, over-encoded name that crams every implementation detail (retry, caching, which replica) into the identifier. It is the counterexample to over-naming. The goal is to reveal the one load-bearing fact a reader needs and let locality carry the rest, rather than stuffing every fact into the name. The lesson it compresses is precision over length, because a name that tries to encode everything ends up both unreadable and another duplicate that can drift, whereas a short name that surfaces the single fact that matters serves the reader better.
XXIII. Time is an input that lies; measure with a monotonic clock, order with logic
Time is an input that lies
The clock feels like a reliable, neutral fact of the universe, but wall-clock time steps backwards under NTP corrections, and it skews between machines, and it can't establish causality between events, so treat it as a hostile input in the sense of tenet IV. Trust the wall clock blindly and your deadlines and ordering and TTLs all break. So in practice you measure elapsed time with a monotonic clock that only ever moves forward, and you establish ordering with logical clocks rather than timestamps, treating time with the same suspicion you'd give untrusted data.
measure with a monotonic clock, order with logic
Use the right kind of clock for each job. Measure durations and deadlines with a monotonic clock, the kind that only ever moves forward and never jumps, and work out the order of events across machines with logical or causal clocks instead of comparing wall-clock timestamps. The reason is that the wall clock lies for both jobs. It can jump backward on an NTP correction or a leap second, so a duration measured against it can come out negative, and a timestamp tells you nothing reliable about which of two events on different machines actually happened first. Picking the clock by the question you're asking - elapsed time, or causal order - is what keeps the timing and ordering logic correct.
happens-before
Lamport's causal-ordering relation: event A happens-before event B only if A could have causally influenced B, say by sending a message that B received. It's stronger and more principled than plain chronological before, because it captures actual causality and not just two timestamps that might have been set by clocks that were never synchronised. Whenever you genuinely need to assert that one event came before another across machines, this is the right thing to reach for in place of a wall-clock comparison, and it's what logical and vector clocks are built to track.
clock skew is a race (IX) wearing a timestamp
Ordering events by comparing wall-clock timestamps across different hosts is the same check-then-act race as tenet IX. It's just dressed up as a time comparison. Clocks on separate machines drift relative to one another, so the order you read off created_at is a guess and not a fact, and two events can show up in the wrong order in exactly the way two interleaving operations can. The timestamp is a costume thrown over a concurrency bug you already know about. The fix is the same family as for any race: order events by a logical clock or a fenced sequence rather than by wall-clock time across hosts.
XXIV. Make the run reproducible; encode the invariant as a test
a bug that escaped is a missing test, not just a bad commit
A bug looks like a coding mistake you fix in the commit that caused it. But an escaped bug also reveals an absent invariant-enforcing test, because something got through and no automated check was guarding the rule. The fix that lasts is the test that re-runs the rule forever and stops the same class of bug coming back, not just the one-line patch to the symptom. Tests are the structural enforcer for invariants that no type can carry, so the real gap an escape exposes is in the safety net, not only in the code that slipped through it.
test the contract, not the internals
Pin your tests to the observable behaviour at boundaries and decision points, the promises the unit makes to its callers, rather than to its private implementation shape. The aim isn't to test less thoroughly but to test at the right level. Tests coupled to internal structure break every time you rename a helper or move the code around, so they turn into a tax that punishes exactly the cheap refactoring the manifesto wants to encourage. Drawing on Kent Beck's test desiderata, contract tests survive refactors because they only fail when the behaviour actually changes.
a result you can't regenerate is an anecdote, not a finding
A result feels like a finding whether or not anyone can reproduce it. But without a pinned environment, versioned data and a recorded random seed it's an unrepeatable story rather than evidence you can build on. Reproducibility is what turns a one-off observation into something you can debug, audit, re-run after a change, and trust enough to act on. An unpinned run that happened to produce a nice number is an anecdote. You can't return to it, you can't vary one input to learn from it, and you can't let anyone else confirm it.
\"it built last quarter\" is not a build
A build that succeeded once is not the same as a build you can rely on. Without a lockfile and a pinned toolchain, last quarter's success was contingent on whatever versions of dependencies, compilers and system libraries happened to be present that day, and none of those are guaranteed to come back. The property that actually matters is determinism: the same inputs producing the same output across different machines and across months. A one-time green build is luck, not a build, because nothing about it can be reproduced on demand.
Works on my machine
The canonical environment-drift idiom names a real failure. Correctness that holds only on your machine is sitting in unpinned dependencies, unrecorded seeds and un-versioned inputs rather than in the structure of the code, so it stops holding the moment someone else runs it. It's offered as a fair defence - it runs fine here, so the code must be correct - but it's the opposite, because a result that depends on your particular machine is unowned, undeclared state and not a property of the program. Reproducibility is the fix: pin the environment so that what works on your machine works on every machine.
XXV. Process is structure when code structure runs out
Process is structure when code structure runs out
After twenty-four tenets pushing correctness down into code, endorsing process sounds like a turn back towards bureaucracy. But some correctness genuinely cannot live in code: who gets paged at 2am, the runbook for a rare failure, four-eyes review on an irreversible action. Where types and tests can't reach, human process becomes the structural enforcer of correctness. The framing treats process as a continuation of the same goal, pushing correctness into structure, rather than a contradiction of it. It carries the load that code cannot, and it should be designed with the same rigour.
a runbook
A written, per-service guide listing the known failure modes and the first response to each, kept for whoever is on call when something breaks. It isn't a book about running. It's the operational playbook you reach for at 3am. The distinction it draws is sharp: observability tells the on-call engineer what broke, but only a runbook tells them what to actually do about a known failure, which is what turns a panicked investigation into a checklist.
one owner of record
A single named, accountable owner designated in advance for every service, feature flag, cache and migration, written down before any incident happens. It means a specific person or team is on the hook, not whoever happens to have touched it last. The value is that who owns this is never the first question asked during an incident, when minutes matter, and it heads off the orphaned service that nobody dares change because no one is responsible for it.
the unowned service is the one nobody dares touch during the outage
A service with no named owner becomes the most dangerous component in an incident, because in the moment when something has to change quickly nobody knows who's authorised to decide, what's safe to touch, or what the side effects will be. It sounds like a remark about team timidity, but it's an argument for ownership-of-record as structural safety. Clear ownership is what keeps a component fixable under pressure, while the absence of an owner turns an otherwise repairable service into an untouchable one at exactly the moment you can least afford the hesitation.
pit of success
A system designed so that the easiest, most natural, default action is also the correct one, so people fall into doing the right thing without having to be careful. The name deliberately inverts pit of failure: instead of a trap that catches the unwary, it's an arrangement where you have to go out of your way to get it wrong. The insight is that correctness lasts when the right thing is the easy thing, so it happens by default and doesn't depend on people staying alert all the time.
Bureaucracy for its own sake wears away the trust it was supposed to encode.
Process is meant to capture hard-won safety. But ceremony applied to cheap, reversible actions just turns into friction, and teams quietly route around friction, and once people are routing around the process they lose the habit of following it in the places where it actually matters. So heavy process imposed without any regard to cost ends up undermining the safety culture it was supposed to build. The bound on process is its own cost. Keep runbooks, approvals and four-eyes review for the things that are genuinely irreversible or critical on-call, and leave the cheap reversible path light, so the heavy controls still mean something when they show up.
The through-line
Every tenet here is one move wearing different clothes
The twenty-five tenets look like separate, unrelated rules. They're not. They're all the same single move applied to different places: pushing correctness out of human memory and into the structure of the system. Types, schemas, ownership, boundaries, atomic steps, deadlines, tests and reproducible runs are all costumes for that one idea. Calling them one move in different clothes is meant to tell the reader what to do with the list. Learn the move underneath, so you can apply it anywhere, instead of memorising twenty-five surface rules and then missing the situations none of them happens to name.
push correctness out of human vigilance and into the structure of the system
This is the one sentence the whole manifesto reduces to. Every tenet is the same move: make the wrong thing impossible or hard to express by leaning on types, schemas, ownership, boundaries, atomic steps and tests, rather than relying on people to remember and apply rules. The reason is blunt. Vigilance gets forgotten under deadline and pressure, so any correctness that depends on someone remembering will eventually lapse, while correctness baked into structure holds for everyone, automatically, for as long as you like. It reads like a motivational summary but it's really the operating instruction behind all the specific tenets.
the wrong thing cannot be expressed in the first place
The goal is to set the system up so that illegal states, hostile inputs, untested irreversible acts and lying clocks just can't be written down: the bad value can't be constructed, the malformed input can't get past the boundary, the destructive action can't fire without its test. Then correctness stops depending on anyone remembering anything. The thing it argues against is the comforting belief that careful, diligent people will avoid the wrong thing. Memory fails under pressure, so making the wrong thing unrepresentable is the only kind of correctness that survives a tired engineer at 2am. And it's a claim about what can be expressed, not a promise that bugs go away.
Make the right thing the easy thing and make the wrong thing hard to even express
Design your systems, APIs and tools so the correct action is the path of least resistance, the thing that just falls out when someone takes the obvious route, while the wrong action is awkward, conspicuous or impossible to write. This is the pit-of-success or affordance principle: correctness should come from the shape of what you build steering behaviour, not from discipline and willpower. It reads like a feel-good slogan, but it's a concrete instruction about how to build interfaces, so that doing the safe thing takes no extra vigilance and doing the unsafe thing means going out of your way.
Write for the person who is going to read this at 2am with a pager going off, and who knows less than you know now
Write the code, the names, the errors and the docs for one specific worst-case reader: a tired, low-context on-call engineer woken by a pager during an incident, who has none of the context you've got in your head while writing it today. The recurring 2am-pager image stands in for the reader the whole manifesto is built around, and it's more than advice about writing docs for the night shift. It's the human version of the objective function. That reader can hold almost nothing in their head, so minimise what they have to hold by putting the necessary context into the structure, the names and the signals, instead of leaving it in your memory.
The Shape of the Whole
Externalise, don't memorise
The companion's one-line slogan, cast deliberately in the same mould as the manifesto's "parse, don't validate". The failures that live between components are too big and too easy to forget to sit in any one person's head, so you push the shape of the whole into an external, consultable model (a dependency graph, a capacity model, a consistency contract) instead of trusting someone to remember it. It's the same move the manifesto makes with types and schemas, just one level up, where the thing being externalised is a model of how the parts compose rather than a property of a single value. The point is that a model a reader can go and look at holds whether or not anyone remembers it, the same way a type does.
Independence is not the resting state of a distributed system
Two components that run on different hosts, in different languages, owned by different teams, feel independent. But that feeling is an assumption, not a fact. They might still share a DNS resolver, a config service, a certificate, or an availability zone, and the moment one of those shared things fails they go down together. Independence between parts of a distributed system is a property you engineer and then verify, not a free starting condition you get for nothing. The law's name is a warning against treating "these are separate services" as if it meant "these fail separately", because the shared substrate underneath usually means it doesn't.
common-mode failure
A common-mode failure is one where several components everyone assumed were independent fail at the same time because they share a single underlying cause: the same DNS, the same config push, the same certificate expiry, the same power feed. The term comes from reliability and safety engineering, where it names the way redundancy can be an illusion, since three redundant systems are worth one if all three lean on the same thing. It's the failure Law I exists to surface. You only see it once you draw the graph of what every "independent" part actually depends on, because by definition no single component's own view shows you the shared cause.
cyclic dependency
A cyclic dependency is a loop in the service graph (A needs B, B needs C, C needs A) that you can't see from inside any single service and that's harmless in steady state, but turns into a deadlock the moment everything starts cold or contends for one resource at once, because nothing in the cycle can make progress until something else in the cycle already has. It's the circular-wait condition from classic deadlock theory, lifted from threads up to whole services. The danger is that it can run for years and nobody notices, since in normal running the caches are warm and the cycle never actually closes, and then it shows itself only on a cold boot or under load, which is exactly when you can least afford it.
metastable failure
A metastable failure is one where a system has two stable states, a healthy one and a collapsed one, and there's a feedback loop that keeps it stuck in the collapsed state even after whatever set it off has gone away. The usual example is a latency blip that sets off a wave of timeouts and retries, and then the retries are themselves the load that keeps the latency going, so taking the blip away does nothing because by now the system is its own cause. The term comes from Bronson and colleagues. It names the most dangerous shape of outage, because the usual instinct - go and find what started it and fix that - doesn't work once the cause and the symptom have changed places.
sustaining feedback loop
This is the loop that keeps a metastable failure going: the system's own response to being overloaded turns into the overload. Retries fire because requests timed out, the retries are new load, that load causes more timeouts, and the wheel keeps turning by itself. To cure it you make the loop's gain less than one, so that each turn produces less load than the one before rather than more. You do that by capping retries, shedding work that's already past its deadline, and de-synchronising clients, so that the response damps the disturbance down instead of feeding it. It's the difference between a control loop that settles and one that runs away, only here it's applied to the behaviour of a whole distributed system and not just a single controller.
hysteresis
Hysteresis is the property that the push which knocked a system into a bad state won't, on its own, lift it back out. The path down and the path up aren't the same, so recovery takes more than just removing whatever caused the trouble in the first place. It's the signature of a metastable failure, and it's borrowed from physics, where a system's state depends on its history and not only on its current inputs. During an outage it shows up as "we fixed the thing that started it and nothing got better". It's the reason recovery from a self-sustaining collapse usually needs you to do something active - shed load, drain the queue, scatter the herd - rather than just put things back the way they were before the fall.
bistable region
A system is bistable when it has two stable operating points, here healthy and collapsed, and it can sit in either one, with an unstable threshold sitting between them. Near full utilisation a system that's perfectly fine at, say, 70% load can flip over into a collapse at 85% that it won't climb back out of on its own, because past the threshold the feedback that keeps the bad state going takes over. The idea comes from queueing theory and dynamical systems. It matters because planning your capacity to the comfortable average hides the cliff. The question isn't "what load do we handle", it's "how close to the tipping point are we actually running", since the region just past it is one transient spike away from collapse.
retry storm
A retry storm is the textbook metastable failure. A dependency slows down or fails, every client retries, the retries pile onto the struggling service as fresh load, that load keeps the failure going, and the storm carries on long after the original fault is gone. What's especially cruel about it is that the congestion feeding the storm usually also blinds the monitoring you'd want to use to find it, so you wind up hunting in the dark for a fire that's feeding on your attempts to reach it. The cure is the manifesto's retry discipline (a budget, exponential backoff, jitter, and a circuit breaker) along with shedding the backlog of already-dead work, so the recovering service doesn't get re-buried straight away.
cache stampede
A cache stampede happens when a popular cached value expires and every request that misses tries to regenerate it all at once, so the regeneration becomes the load that causes the next miss, and the hot key ends up melting the backend it was supposed to protect. It's a metastable loop in miniature. The fix is to coalesce the work, so that one regeneration gets computed and shared by all the waiters, rather than each missed request going off and recomputing the same thing on its own. That turns a synchronised thundering-herd spike back into a single unit of work.
distributed invariant
A distributed invariant is a rule that has to hold across three or more parties where no single party can see far enough to enforce it on its own: "this seat is sold at most once" across many booking nodes, "this account never goes below zero" when the debit and the limit live in different stores, "an order is shipped or refunded but never both" when two workflows act without knowing about each other. The manifesto's defences against contradictory state are all dyadic - a value, an owner and its derived views, a single hand-off - so they don't reach this case. What the law demands is that you choose, out loud, where such a rule lives (a coordinator, a saga, a quorum, or an explicit reconciliation), because the alternative is an invariant held up only by every service happening to behave itself, and that's the swallowed error written at the scale of the whole system.
coordinator
A coordinator is a single component that serialises a decision so that a rule spanning many parties gets enforced in one place: everyone asks it, it decides one at a time, and the invariant holds because there's exactly one authority. It's one of the honest homes for a distributed invariant, and it's also the law's sharpest warning, because the coordinator you add to enforce consistency is itself a fresh single point of failure and a new shared substrate (Law I). You have to pay for it, and it's only worth it where the invariant is real and breaking it costs more than the coupling the coordinator drags in.
saga
A saga implements an operation that spans several services as a sequence of local steps, each one with a compensating action that undoes it if a later step fails, so the whole thing reaches either a complete success or a clean, compensated rollback without one distributed transaction holding locks across all of them. It's Garcia-Molina and Salem's pattern for long-lived transactions, and it's the usual answer to "create the order, charge the card, reserve the stock" as one invariant wearing three services. Its cost is honest: the compensations are themselves irreversible-decision code that can fail halfway through, so a saga trades the coupling of a distributed transaction for the burden of getting every undo right.
quorum
A quorum requires a majority of nodes to agree before any action is taken, so that a single cross-party fact (who holds the lock, what the committed value is) has one answer that survives individual failures. Consensus algorithms like Paxos and Raft turn this into a primitive you can build distributed invariants on. The trade is the one CAP names: a quorum buys you agreement at the price of availability, because when the network partitions and a majority can't be assembled the system has to refuse to act rather than risk two answers, so you keep this heavy machinery for the facts that genuinely can't tolerate divergence.
eventual consistency
Eventual consistency is the honest admission that a fact is allowed to be temporarily disagreed-upon across parties, paired with the promise that the copies will converge once updates stop, usually backed by a reconciliation sweep that finds and repairs the divergence after the fact. It's the cheap, available alternative to coordinating every change, and the law's point is that most rules people reflexively call invariants will tolerate it. You keep the strong, coordinated machinery for money, safety and the genuinely unrepeatable, and you let everything else converge. The danger is leaving the divergence undeclared, because an eventual-consistency design with no named reconciliation point is just an unbounded inconsistency wearing an optimistic name.
the CAP bargain
CAP is the result that a distributed system facing a network partition can have at most one of strong consistency and availability, not both. When the nodes cannot talk to each other you either refuse to answer, which keeps one consistent truth, or you answer from each side, which keeps you available but lets the two sides drift apart. Brewer conjectured it and Gilbert and Lynch proved it. It is the trade the manifesto names and defers, and the companion makes it concrete: a coordinator or quorum that you pick to enforce an invariant has, just by being picked, already chosen consistency over availability for the moment the network splits. That is a cost you sign up for on purpose, not something you stumble on during the incident.
control loop
A control loop is any component that reads a signal, checks it against a target, and acts to close the gap. An autoscaler watching load, a circuit breaker watching errors, a load balancer watching health, a cache watching demand. Each one is a controller in the control-theory sense, and taken on its own each can be correct and stable. What the law worries about is what happens when several of them act on overlapping signals inside one system. The interaction is a behaviour none of them specifies and none of them can see, so a controller that is provably stable by itself tells you next to nothing about how it will behave once it has company.
coupled-loop oscillation
Coupled-loop oscillation is what you get when two or more control loops act on the same variable on similar timescales, each one chasing a signal that the other is busy moving. Together they hunt and flap in a place where each was supposed to settle down. The autoscaler removes capacity because the breaker shed load, the breaker opens because the survivors saturated, and the correction just makes the fault worse. Two loops that are each stable in isolation can compose into one unstable loop the moment their timescales line up. The control-theory cures are either to separate their timescales, so the fast one settles before the slow one stirs, or to give one loop authority over the shared lever and make the others defer to it. You pay for that with the damping, and the lost responsiveness, that keeps them from fighting.
gray failure
A gray failure is a degradation that is bad enough to break the end-to-end journey but too mild to trip any single component's health check. Every service reports green while the request that has to cross all of them does not arrive. The term is Huang and colleagues', and the idea at the heart of it is differential observability: the system's own view of its health and the user's view have come apart. It is why you cannot add up green signals into a true picture of the whole. Catching it needs a probe that walks the entire path the way a user does, not a sum of per-component checks that are each answering the wrong question.
differential observability
Differential observability is the gap between what a component sees about itself and what someone on the outside sees about it, whether that outsider is a user or a whole-path probe. The node passes its own liveness check while quietly dropping one packet in twenty. To its monitor it looks healthy, and to everything routed through it it is a slow poison. This is the mechanism behind gray failure, and it is why component health and system health are different questions. The fix is to measure from the outside, end to end and against a budget, because only a witness that sees the whole journey can spot a failure that each node, judging itself, honestly reports as fine.
fault injection
Fault injection means deliberately putting failures into a real system, killing a node or adding latency or dropping packets or severing a dependency, so you can watch how the whole thing behaves. The point is to make the emergent, composition-level failures happen in daylight with someone watching, rather than at three in the morning with no one around. A "game day" is a rehearsed exercise of this run at the scope of the whole system, and "chaos engineering" is the discipline of doing it continuously and on purpose. The catch the law names is that the probes are themselves load and risk. A game day is a controlled outage, and synthetic traffic is real traffic the system has to carry. So you bound them, and you only run the drills you are actually going to act on.
end-to-end probe
An end-to-end probe is a synthetic check that exercises the entire path a real request travels, edge to edge, against a deadline, instead of checking each component on its own. It is the only thing that catches a failure living in the composition of behaviours, the eight services each adding a bit of tail latency that is harmless alone and fatal once they stack, because no single node is watching the sum. The discipline is to verify at the scale of the failure. Component health answers "is each part alive", and only a whole-path probe answers "does the system do its job". But a probe nobody reads is just a silent fallback with a dashboard for a disguise, so it has to be watched and acted on like any other signal.
normal accident
A normal accident is a system-level failure that comes out of the interaction of components that are each working exactly as designed, not out of any one broken part. In a tightly coupled, interactively complex system, accidents like this are not freak events. They are a normal, expected property of the structure. The term is Charles Perrow's, and it is the intellectual backbone of the whole companion. It is why "every part is innocent" and "the whole still failed" are not a contradiction, and why some failures can only be reasoned about, and contained, at the level of the composition, never at the level of the part.
From the phase guides
encode the whole investigation in a slug
A slug is the short URL or title identifier for an issue, the terse fragment that names it. You cannot cram an entire investigation and its analysis into that brief string, so the title should instead surface the one load-bearing risk. Newcomers may not know slug as the web and CMS term for a compressed title fragment, and might picture the animal. The guidance is that an issue title's job is to show the single most important risk at a glance, not to summarise the whole diagnosis.
a rumour with a screenshot
A bug report that carries some surface evidence, a screenshot, but no deterministic way to reproduce the problem, so you cannot act on it as a sized piece of work. The screenshot feels authoritative, but a picture proves nothing about the cause or the scope. It shows that something happened once, not what triggered it or how often. Without an on-demand repro the report is hearsay dressed as fact, and it has to rank below anything you can actually demonstrate, because you cannot tell whether a candidate fix worked or whether the symptom just happened not to recur.
a zombie ticket that rots in the queue
An issue that got no explicit disposition during triage. It was neither picked up to be worked nor deliberately closed, so it lingers undead in the backlog, silting up the queue and stealing attention from every future triage pass. This is the practical form of tenet XIII's silent swallow applied to decisions: a swallowed non-decision leaves the ticket in a limbo that is neither alive nor dead, and that ambiguity keeps costing you. Every ticket should leave triage with a clear state, because the absence of a decision is itself a decision to let the queue rot.
wire it in dark
Ship a new integration switched off, with the old path still carrying the traffic, then turn it on for a sliver of requests and ramp from there, so the first contact the new dependency has with production is at one per cent and with a way back already in place. The phrase borrows from a dark launch: the code is deployed and exercised before it is trusted, rather than cut in for everyone in one irreversible move. Wiring in dark makes the fall-back the default and forces the new edge to earn its traffic, so a contract bug or a latency surprise shows up as an incident at one per cent instead of an outage at a hundred. It is the integration-phase form of shipping reversibly (tenet XX): the edge that is wrong about something, and it always is about something, becomes a toggle you flip rather than a hotfix you have to survive.
a vow to be perfect at the other side's worst moment
What an edge with no fall-back actually promises. When you call another service and have no degraded path for its absence, you have committed your side to behaving correctly at precisely the moment the other side is failing, which is the moment it is least able to hold up its end. A dependency you cannot do without is a dependency whose worst day becomes your worst day. The cure is to decide the fall-back in advance (tenet XIX): serve stale, serve partial, or serve the old path, so the far side going down costs you a feature and not the whole request, and you are never leaning on a promise that comes due exactly when it cannot be kept.
the demo worked, the integration didn't
The gap between a component passing on its own and the composition working end to end. A part can sail through its own tests, its own demo, and its own green dashboard and still fail the instant it becomes one hop in a longer path, because the failure lives between the parts and not inside any of them (the companion's fifth law). The demo proves the node; only a request sent the whole way along the route proves the system. If you have not walked the path a user walks, end to end and against a deadline, all you know is that each part is alive, which is a weaker claim than the one that matters, that the system does its job.
the adapter that quietly became load-bearing
A throwaway shim, written to bridge two systems during a migration, that is still in the critical path long after the migration is forgotten, owned by nobody and guarded by a contract test that went flaky and got skipped. Temporary integration code has a way of outliving its reason and hardening into permanent infrastructure no one dares touch, so that when it finally breaks the first question of the incident is who owns this. The defence is an owner of record and a sunset stapled to every edge the day you add it (tenets XXV, XIV), with a contract test the owner keeps green, so the boundary it guards never decays back into folklore.
noise dressed as a finding
Any conclusion you draw from a flaky, non-deterministic reproduction, where the randomness in the repro is passing itself off as a real result. If the bug only sometimes shows up, you genuinely cannot tell a fix from a coincidence. The symptom going quiet might mean you solved it, or it might just be the dice landing differently. A flaky repro is not a weaker version of a solid one. It is a different and harder problem, because every observation you make on top of it inherits that randomness, so the first job is to make the repro deterministic before you trust anything it tells you.
a guess wearing a lab coat
A scattershot change made on the off-chance it helps, dressed up to look like methodical debugging when really it tests nothing and rules nothing out. The lab coat is just costume. Changing things and watching whether the symptom moves is still guessing, unless you first say what result would prove your hypothesis wrong and then bisect toward it. Real investigation narrows the field of possible causes at every step. A guess in a lab coat leaves that field exactly as wide as it was before, even when the symptom happens to go away, because you never found out why.
hope does not converge
"Converge" comes from binary search, where each well-chosen test should halve the remaining possibilities and close in on the cause. A theory you can't falsify - one you cannot design a test to kill - doesn't narrow that field at all, so however many hopeful pokes you take at the problem, none of them get you anywhere. Only a hypothesis you can falsify lets you knock out half the unknowns each step, the way bisection does. The phrase isn't pessimism dressed up as a swipe at optimism. It's a flat statement that hope tests nothing, so it never homes in on the bug.
Cash in the observability you built
Before you add any new probe or log line to an investigation, spend the traces, metrics, and logs you already emitted by actually reading them first. The idea is to treat earlier instrumentation as a prepaid deposit. You paid the cost of emitting it up front, and the investigation is when you redeem it, instead of reaching straight for new instrumentation and guesswork. Observability is prepaid debugging. Cashing it in first is faster, and it's more honest than bolting on fresh probes when you haven't even used up what you already have.
a race condition in your reasoning
A causal story you put together by sorting events on wall-clock timestamps gathered across several machines, whose clocks skew and drift, so the order you reconstructed may not be the order things actually happened. The same clock-skew that causes real race conditions in distributed systems also corrupts the analyst's mental timeline, so a timestamp-sorted run of events is itself a bug in your thinking, not a sound basis for blame. It turns tenet XXIII inward. Distrust cross-machine ordering in your own analysis exactly as you would distrust it in the system you're testing, and anchor causality on logical ordering or shared identifiers instead.
folklore the next person re-derives from scratch
A diagnosis that survives only in someone's head, or in a chat thread, rather than as an owned, citable record with an explicit list of what got ruled out. Like oral folklore, knowledge that lives only by being retold degrades a little each time it's passed on, and in the end the next investigator has to re-walk the whole maze you already cleared. Make the diagnosis structural - written down, attributed, with the ruled-out lines included - and the cost of the dig is paid once instead of being quietly re-incurred by every person who hits the same problem later.
a hope wearing a green tick
A fix you've declared confirmed only by re-reading your own diff, or by watching a check go green, when that check never once watched the running system show the corrected behaviour. The green tick is the costume of proof. A test that was never red against the real bug proves nothing about whether the bug is gone. All it proves is that the test passes. Verification is a dynamic claim about a system in motion, so confirmation has to come from watching the running system, not from static confidence in code you wrote, or a check that happened to be green before and after for reasons that have nothing to do with the bug.
A green check that was never red proves nothing
A test that passes on the fixed code but was never seen failing on the broken code gives you no evidence that it actually exercises the bug. The red-then-green transition is what gives a passing test its meaning. Watching it fail for the right reason first proves the test reaches the defect, and only then does the later pass prove the fix works. Without that, all you've confirmed is that your check agrees with your patch, which it would do even if the check were looking at the wrong thing entirely. This isn't a claim that passing tests are worthless. It's a requirement that you verify the test can fail before you trust that it passes.
manufacturing concurrency to verify it is theatre
Going through elaborate verification motions that don't actually test a real risk, like fabricating concurrent execution to test code that genuinely only ever runs once. Like security theatre, it performs diligence without cutting any real risk, spending effort and attention to look thorough rather than to catch a defect that could plausibly happen. Replay, interleaving, and concurrency tests are worth their cost only where two actors can really meet in production. Staging that collision for code that by its nature only runs once is effort spent on a threat that cannot happen.
a verification that lives only in your terminal history
A one-off manual confirmation that was never pinned down - its inputs, seed, and environment unrecorded - and never turned into an automated regression test, so it protects no one once this release ships. A pass like that is an anecdote. You can't regenerate it, and a check that isn't automated is one the next change will quietly undo with nobody noticing the protection has gone. This compresses tenet XXIV. A verification only earns its keep if it's reproducible and lives somewhere the next change is forced to re-run it, not as a throwaway command buried in your shell history.
don't instrument theatre you will never read
Emitting metrics, dashboards, or logs that no decision is ever actually gated on - telemetry that exists to look diligent rather than to be read and acted on. It echoes security theatre, and tenet XVIII's point that noise is its own failure, because unread signal isn't free. It adds cost, it clutters the views you do read, and it dilutes the things that matter. Telemetry earns its place only when some real decision - a promotion, a rollback, an alert - genuinely consumes it. If nothing reads it, it's a prop, and you shouldn't have built it.
An unbounded retry is the queue that melts the fleet, only it is wearing a helpful name
Retrying sounds like a benign, helpful resilience feature, but a retry without backoff, jitter, and a circuit breaker is the very same unbounded-queue failure mode as in tenets V and VII, just disguised under a friendly name. The comforting word retry hides the fact that it amplifies a transient blip into a thundering herd that hammers the dependency, takes it down, and then takes the fleet down with it. The reframe is to see unbounded retry for what it is, a queue with no admission control, and to bound it the way you'd bound any other unbounded resource.
mistook \"process gone\" for \"work done\"
A process exiting is not the same as the work it was responsible for being finished. If a process terminates without draining its in-flight obligations, it drops requests that are still being handled and loses work that was received but not yet acknowledged or persisted. The phrase names the conflation behind dropped-request deploys. People read the absence of the process as completion, when graceful shutdown is itself the last piece of work the process owes (tenet XII), and a missing process is not the same thing as durability. The fix is to drain and finish, or durably hand off, before exiting.
the planning you didn't do arriving with interest
Improvising at 3am during an incident is the deferred cost of the runbook, ownership, and procedures you never wrote, now coming due at the worst possible moment and compounded by pressure. The debt-and-interest metaphor is exact. Skipped preparation is a loan, and you pay it back at the very point where the stakes are highest and you have the least context to work with and the cost of working it out live is at its steepest. So the lesson is to write procedures for foreseeable failures up front, while it is cheap and nobody is panicking, instead of borrowing against some future incident that will charge you interest in stress and downtime.
keeps calling the old number long after you hung up
After you delete a feature or endpoint, callers and dependencies carry on hitting it anyway. You control your side of the contract but not theirs, and the rest of the world does not update the moment you do. It is like a disconnected phone line that people keep dialling - the removed thing still gets traffic from clients that never learned it was gone. This is what makes removal uniquely dangerous. The claim that nothing uses something anymore can't be proven from your side alone, so safe deletion needs telemetry that actually watches for those lingering callers before and after you pull it.
The reference you didn't search for is the page at 2am
Here a page is a pager alert, not a sheet of paper. The dependency on a thing you are about to remove that you failed to find in your search becomes the on-call incident that wakes you in the night. An incomplete dependency search doesn't make the missed caller go away. It just puts off the discovery until the worst moment, when that caller breaks in production. The wordplay leans on the pager sense of page, and the point is that how thorough your dependency search was converts straight into how likely you are to get woken by what you overlooked.
a sample that misses every other consumer
Searching only your main repository for callers of something you want to remove gives you a biased, partial sample of the real dependency graph. It leaves out other teams' services, dashboards, scheduled jobs, third-party integrations and stale clients in a systematic way, none of which you can see from where you are standing. Framing "I grepped the main repo and found nothing" as sampling error is the point here, because the consumers that are missing from your sample are exactly the ones outside your repo, and that is precisely where the surprises live. So removal has to be gated on live telemetry showing zero real usage, not on a code search you happen to be able to run.
a date you detonate
A removal pinned to a fixed cut-off date that breaks every consumer who hasn't migrated the instant it arrives, set against the alternative of an extendable deprecation state that lets stragglers drain off over time. A detonation has no losing move. There is nothing you can do once the date hits except sit through the breakage, whereas an extendable state keeps a path open for the last few callers. The phrase puts a date you detonate up against a state you can extend, and the lesson is to go for the state that degrades gracefully rather than the deadline that explodes.
the old thing with an apology attached
This sounds like a polite, considerate form of deprecation, the responsible thing to do, but an indefinite deprecation window with no real removal date works out the same as keeping the old behaviour forever, and the deprecated label changes nothing for anyone. It inverts the idea that announcing deprecation is progress. Without a dated, owned sunset you have just attached a warning to permanent debt, not removed anything. Real retirement needs a committed date and an owner. Otherwise the apology is the only thing that ever ships.
trust the silence
This sounds like believing that quiet means consent, the unsafe assumption that no objection equals agreement, but here it means letting a thing sit with reads diverted and writes stopped long enough that the continued absence of traffic becomes reliable evidence it is safe to drop. It inverts the usual warning that absence of signal is not evidence. After a full usage cycle has gone by with no demand, observed silence becomes positive proof rather than just a lack of data. The key is sizing the cold window to the actual usage cycle, monthly or quarterly jobs included, and not to some arbitrary stretch of the calendar.
answering queries with a ghost
A cache, index, or replica that outlives the source of truth it derived from carries on serving data for a record that no longer exists, returning answers from something that is effectively dead. It dramatises tenet XIV. If you delete the owning record but leave its derivations in place, those copies turn into liars, confidently returning stale data for an entity that has been removed. The fix is ordering. The derived data has to be deleted or re-homed onto a live owner first, because a derivation with no backing source of truth is a haunted answer that looks authoritative and is wrong.
an \"archive\" that quietly becomes a permanent copy nobody is allowed to forget
An archive sounds like a safe, responsible place to store retired data, but calling something an archive to dodge true deletion keeps regulated data on disk indefinitely, where it stays breach surface and legal liability rather than being genuinely retired. The reassuring word archive disguises non-deletion. For duty-bound data the archive is just the data you were obliged to destroy, still sitting there. Proper retirement of such data means an enforced TTL with a real hard-delete path, not a renamed bucket that nobody is ever allowed to empty.
nothing left behind names the absence
After you remove something, no dead code, zombie feature flag, stale config key, dashboard panel or doc should still mention the thing that is gone, because a name that points at something no longer there misleads a future reader worse than saying nothing at all. This ties tenet XXII (a misleading name is worse than no name) to the act of subtraction. A half-finished removal leaves a trap, since the lingering reference tells the next person the thing still exists and they will go on to reason and act on a falsehood. A clean retirement takes out the references along with the implementation.
Watch for the resurrection
A removed thing has a strong tendency to come back to life. A revert restores a deleted flag, a migration copied from an old project recreates a dropped table, a new feature quietly re-adds the endpoint you retired. The framing treats retired as a state you have to actively defend rather than a one-time act that stays done on its own. The durable defence is to encode the absence as an invariant, an alert or test that fails if the thing reappears, so that any accidental resurrection trips a wire instead of silently undoing the removal.
the most confident voice in the room rather than the most correct one
A retrospective run from memory drifts towards whoever states their version most confidently, and confidence has nothing to do with being right. Human recollection actively selects for assertiveness, so without an artefact to settle the matter (a reproduction, a log, a trace) the loudest hypothesis wins regardless of whether it matches what actually happened, and the team learns the wrong lesson. It sounds like a note about meeting dynamics, but it is really an argument for grounding retros in evidence. Only something you can re-run or point at can settle which account is true, rather than which account got delivered with the most certainty.
blameless by construction
Blamelessness that you build into the structure, by pointing every retrospective finding at the system gap that let the failure through and not at the person who happened to trip over it, rather than hoping people will choose to be polite on the day. The phrase echoes correct-by-construction. You build correctness into the structure instead of hoping for vigilance, and in the same way you build blamelessness into how the questions get framed. It has to be structural because blame-hunting just teaches people to hide their failures, and that starves you of the very information a retrospective is supposed to bring out, so the cure needs baking into the process rather than left to goodwill.
spent signal and bought noise
Add an alert that fires on every single deploy and you train the team to ignore it and then mute it, so you have spent down the team's finite attention, the signal budget, and got back nothing but noise. It treats attention as a budget in the spirit of tenet XXI. A noisy alert is a negative-return purchase because it costs real attention and it chips away at trust in every other alert sitting near it. So the corollary is that you pay for new alerts out of that same budget by deleting a stale one whenever you add one, which keeps the total noise bounded and keeps whatever signal you do emit meaning something.
process with no structure behind it
Process and structure can sound like the same word, so this reads as a contradiction at first, but a retrospective that ends in unowned, undated good intentions is pure ceremony and changes nothing, while real process has to carry load through a named owner, a date, and a tracked issue. It plays on tenet XXV, that process is structure when code structure runs out. A slide full of intentions is process in name only, with none of the load-bearing structure that would actually make the change happen. The test is whether anyone is on the hook by when, not whether a meeting happened.
a sentence with no one on the hook and no clock running
A retro follow-up like we should add a test that names no owner and sets no due date, so it sits there unaccountable and unscheduled and never actually ships. On the hook means a single accountable owner, and clock running means a real expiry or deadline, and without both of those the item is theatre that evaporates the moment the meeting ends. Contrast it with a proper action, which is a tracked issue with one named person and a date, because a follow-up that nobody is responsible for and that is due at no particular time looks exactly the same as a follow-up that was never made at all.
turned the loop back into a line
A retrospective whose only output is a write-up ends the development lifecycle in a document instead of feeding triage-ready work back round to the start, which breaks what is meant to be a cycle. The lifecycle is supposed to be a loop. A retro's findings should re-enter triage as owned, actionable issues, so the circle closes and the next iteration comes out better. A write-up with no filed issues is a dead end, a line that stops rather than a loop that keeps going, so the real deliverable of a retro is owned, triage-ready work and not prose that nobody is ever going to turn into change.
For the principles these compress, see MANIFESTO.md; for how they play out, see the guides in howto/.