We Forgot To Check

Much of the Internet was down last year - 20th Oct 2025. Amazon published an analysis, but don’t bother reading it, I can summarise it for you and tell you what we can learn from this.

We have multiple copies of a key program (the “DNSEnactor”) all running at once (like the Space Shuttle), so if one fails then others will take up the slack. But we forgot to check what happens when one of them hasn’t failed, but is just running really slowly. When that happened on Monday, the other, identical programs overwrote what the slow one did, and this caused the system to delete a key “DNS record” that the DNSEnactors would normally keep up to date. This record is used throughout our vast estate of many programs, including the ones that let programmers run software “in the cloud” (i.e. on our virtualised servers). That’s why Slack and Fortnite and many others had a bad day.

Thoughts:

1. This isn’t a root-cause analysis, or even trying to be one. As usual, there are no people in Amazon’s report, so it’s very hard to tell whether, for example, the company’s famously frugal culture contributed to the missed check. (Any ex-Amazonians here want to comment?)

2. I’m sure there’s a six-pager floating around Seattle with a lot more about what failed in the process and how to unleash an army of Chaos Monkeys to catch more such latent bugs.

3. Despite all the wailing and gnashing of teeth, Amazon has excellent uptime: massive, whole-region outages like this do occur, but only every few years, which is six nines or so and very, very good.

4. This is a timely reminder that humans know how to make sure not to forget things, but we’re also very good at forgetting to put in place the anti-forgetfulness measures. That’s why unit tests and certifications and third-party reviews are so helpful, because many eyes make bugs (and missing processes) shallow.

What are you doing so you don’t forget not to forget?