The Postmortem That Learned Nothing
A retrospective that ends in a document has produced a story. The only one worth the hour is the one that produces something that re-runs without you.
Slack went down for hours in February 2022, and the way it happened is worth sitting with. They were in the middle of a routine upgrade to the cache fleet. The small programmes that keep track of which servers are alive restarted on about a quarter of the machines. Every time one of them restarted it briefly wiped that machine's slice of the cache. That's fine on its own. Emptied caches just refill from the database, and the system is built to do exactly that. The catch was that one of the queries doing the refilling was the expensive sort, the kind that has to reach across every shard to answer one question. There was no bug anywhere in this. The upgrade was routine, the cache refill was working as designed, and the query had been quietly expensive for ages without anyone minding. But it was peak hour, and the upgrade and the wiped caches and that costly query all landed inside the same few minutes. Each one was harmless by itself. Together they buried the database and messaging was down for hours. So when the team came to write it up they hit the question that every honest postmortem hits, which is: which of these was the cause? And the truth is all of them, and none of them, and really the question doesn't fit.
The idea that there's one root cause to find, we got that from manufacturing, and it doesn't make the trip across very well. Take the Five Whys, where you ask why five times until you land on the one broken thing. Toyota built that for a factory. In a factory a defect often really does come back to one worn die or one step somebody skipped, so it works. Software isn't like that. Software fails the way Slack failed, which is when several perfectly reasonable things, each safe on its own, happen to line up at the wrong moment. Ask "why did it break" of something like that and you'll get an answer, sure, but the answer is just whichever thread you decided to pull on. Richard Cook spent years studying catastrophe in hospitals and other places that kill people when they go wrong, and he was blunt about it: in a complex system there isn't a single root cause, there's a set of contributing factors, each one necessary and none of them enough on its own, that happened to come true at the same time. Hunting for the one cause isn't investigation. It's wanting a tidy story and calling that rigour.
Which is why the document is the wrong thing to produce. You write a page of narrative, you name a neat root cause, you list some action items, and what you've made is a story you tell yourself so the whole thing feels less frightening. Stories don't stop the next outage. What a retrospective is actually for is changing the system so the same failure is harder to build next time. A test that re-runs the exact failure on every commit and goes red the moment it comes back. An alarm that fires on the early shape of the cascade before it's a cascade. A setting that just can't be flipped the dangerous way any more. And that difference matters more than almost anything. A document needs the next engineer to have read it and to remember it on the day. A test needs nothing from them. It just won't let the broken thing back through, whether or not they've ever heard of the incident. The document is leaning on somebody remembering, and the test isn't leaning on anyone at all, which is why a test is still doing its job long after the people who were there have moved on.
After a cascade the reflex is to bolt on a guard, some check or limit that'll catch it earlier. Sometimes that genuinely is the only thing you can do. But a guard is one more thing you have to deploy and maintain and trust, and the better outcome, when you can actually get to it, is to take the failure mode away rather than stand a sentry in front of it. Did that scatter query need to exist in the first place? Could the data be laid out so the question never has to cross every shard at all? An outage you've designed out can't come back. An outage you've only guarded against is sitting there waiting for the guard to lapse.
And this is where most retrospectives quietly die. The meeting wraps up, the action items get written down, and they go into the backlog, into some column nobody's measured on, where they sit next to features that have an owner and a deadline and so always win the sprint. Nobody closes them, because closing them means admitting out loud they're never getting done. Nobody starts them either, because there's always something louder going on. So they sit, and they age, and after a while they're just part of the furniture. The team picks up the lesson that's really being taught, which is that a postmortem is a ritual you go through and then walk away from, that the document is theatre, and that the system is no safer than it was the day before you all sat down. Then the next engineer walks into the same trap, because somebody wrote the trap up carefully and nobody ever removed it.
So the discipline is to refuse to let the lesson live in a document. The failure turns into a failing test that blocks a deploy. The vulnerability turns into a deleted setting that can't be switched back on. The learning turns into an alarm that'll fire again if the pattern comes back. And the work that honestly can't be done today goes into the same queue as everything else, with an owner and a date on it like real work, because an action item nobody can schedule is one nobody's going to do, and one nobody's going to do was only ever a story. A retrospective isn't really there to explain the outage. It's where you change the system so the next person can't build the same outage over again. If nothing about the system is different on the Monday after the meeting, then whatever you held it for, it wasn't learning.
In the manifesto, this is tenets (XXIV) and (XXI).
Sources
- [Cook 1998] Richard I. Cook, "How Complex Systems Fail". Cognitive Technologies Laboratory, University of Chicago, 1998. https://how.complexsystems.fail/. No isolable root cause: failure needs multiple factors, each necessary and none sufficient alone; tenets XXIV, XXV.
- [Ohno 1988] Taiichi Ohno, "Toyota Production System: Beyond Large-Scale Production". Productivity Press, 1988. https://www.routledge.com/9780915299140. The Five Whys as a single-cause manufacturing technique, the inheritance the post pushes back on; tenet XXIV.
- [Slack 2022] Slack Engineering, "Slack's Incident on 2-22-22". Slack Engineering, 2022. https://slack.engineering/slacks-incident-on-2-22-22/. The Consul-restart, emptied-cache, scatter-query cascade at peak as jointly sufficient factors, not one root; tenets XXIV, XXI.
One of a series of field notes on building software for the way minds actually work: tired, distractible, ordinary, and now partly machine. They all lead back to the manifesto behind them, The Shape of the System.