The Shape of the System

Hostile Until Proven Otherwise

Every value that crosses into your code arrives wearing a disguise, and the dangerous ones are dressed impeccably.

implementing reviewing triage

Picture a doorman who checks the uniform and not much else. A man shows up at the service entrance in the right jacket, with the right clipboard, and he says the words you'd expect, so the doorman waves him in. That is what the jacket is for. It saves the doorman the trouble of working out who the man actually is. A disguise is built to beat the one check you've decided to make, and software does the same thing all day. It looks at the shape of some value that has just arrived from outside, sees that the shape is correct, and then it hands that value the run of the place. The question it never asks is the one that matters. Not "is this well-formed" but "should this be allowed to do the thing it is about to do". The person who robs you isn't wearing the wrong clothes. His clothes were fine. That was the trouble, because the clothes were all anyone looked at. A value can pass every test you wrote and still be a weapon. Well-formed is one question. Whether the thing is harmless is a completely separate one, and you went and asked only the first.

There's code you control, and there's data crossing into it from somewhere you don't, and that line is what this whole thing is about. The instinct is to check the shape at the line and then unclench, as if a correct form were some kind of character reference. It isn't. The right way to stand at that border is not trust, and it isn't even trust-but-verify. It's hostile until proven otherwise. Assume the thing arriving is an attack and make it earn its way in. That's not paranoia. It's just refusing to let a costume stand in for an identity. There are three disguises worth being able to recognise on sight, and then a fourth door you forgot you'd left open.

The first disguise doesn't break any rules. The input is legal in every way you can name. It's just enormous, or, worse than that, it's tiny and it unfolds into something enormous the second you start to work on it. The neat example here is the thing called the billion laughs. It's a scrap of XML, under a kilobyte, perfectly well-formed: ten little definitions, and each one refers ten times to the one below it. Nothing about it is malformed. But hand it to a parser that isn't guarding itself and that kilobyte becomes a billion copies of a single word, around three gigabytes of memory, and the machine drops to its knees having been asked something it agreed was valid. The validity was the weapon. The fix is to bound the size before you commit to it rather than after. Put a cap on the input. Put a cap on how far it's allowed to expand, and on how deep it can go, and settle up front on the most you're ever willing to spend, because otherwise you allocate on the strength of a number that some stranger picked for you.

That number a stranger picked is the heart of the worst one. In 2014 the whole web learned the name Heartbleed, and underneath the name it was nothing fancier than a program believing a length it had been told. There's a small part of the protocol that secures connections that lets one side say, roughly, "send this back to me, and it is sixty-four kilobytes long". The widely used library that handled that copied back the stated number of bytes and never checked whether the thing it had actually received was that long. So an attacker would send a single byte, swear blind it was sixty-four thousand, and the server would hand back sixty-four thousand bytes of whatever happened to be sitting next to that request in memory. Other people's passwords. Session cookies. Often enough the server's own private keys. One missing check, on a length the other side supplied. The envelope tells you how much is inside it, and you shouldn't believe a word of that either.

The second disguise is a value that's inert data in one place and a live instruction in the next. The classic case is the database, and a cartoon turned it into folklore: a mother names her son Robert'); DROP TABLE Students;-- and when the school's software glues that name straight into a database command, the name stops being a name. It becomes an order to delete every student record. It's a joke on a coffee mug and also a working attack. In 2015 the British telecoms company TalkTalk lost the details of nearly a hundred and fifty-seven thousand customers through exactly this, an attack so old and so well understood that the only surprising part was that anyone still had the door open. The pages that let it in were forgotten legacy ones nobody was watching. The fine came to four hundred thousand pounds. Log4Shell, late in 2021, was the same trick one floor up. A string that a server only wrote to its logs, something as harmless-looking as the browser name a visitor sends along with a request, got quietly read by the logging library as an instruction to go off and fetch a chunk of code from an attacker's server and run it. The bug is always shaped the same way. There's a place where you thought you were handling text and the layer underneath you thought it was handling commands. The defence isn't to "sanitise" in some vague hopeful spirit. It's to keep data as data wherever it crosses a boundary, escaped so it can never be read as part of the command around it.

The third disguise is the oldest one, and the most trusted: the request simply tells you who it is, and you take its word. A user number in the address bar. A role in a cookie. A field in the body saying isAdmin is true. A header announcing which customer this is. None of that is an identity. Each one is a claim of identity, and a claim is exactly the thing an attacker gets to write for himself. It's the jacket again. The discipline is to never act on who a request says it is. Work out the identity from something you can actually check, a token you signed, a session you handed out, and then re-check what that identity is allowed to do at the moment it tries to do it, on your side of the wall, every single time. Claimed authority is a master key, and people leave it lying about on the counter.

And here's the door you forgot. It's tempting to think the boundary is the edge of the network, the place the strangers come in from, but it isn't. The package you imported is code you are shipping without having read it, and it runs with all of your privileges. In 2018 a popular little JavaScript library called event-stream changed hands, because the maintainer was tired and glad of a volunteer, and the volunteer, who had only asked nicely, slipped in code to empty people's cryptocurrency wallets. It was downloaded millions of times before anyone read what it now did. That's the guest in the perfect jacket, only mass-produced. And the pipeline that builds and signs your software is an input too. In the 2020 SolarWinds attack the malicious code went into the build itself, so the poisoned update left the door wearing the vendor's own digital signature, trusted all the way, and reached something like eighteen thousand organisations. The cure is the same posture, just turned to face inward. Pin the exact versions. Verify what you pull in. And the machine that builds your code should be treated as if it were production, because it runs with your secrets.

Underneath all four faces there's one move. Stop asking whether a thing is well-formed. Ask instead what the worst is that it's allowed to do, and whether it's earned the right to do it. Hostile until proven otherwise won't hand a judgement off to a costume, and it holds no one in contempt. The boundary is the one place where you get to make that decision cheaply, with a clear head, before anything has actually happened. Once a value is inside, it's already wearing your colours, and everyone downstream salutes it.

In the manifesto, this is tenets (IV) and (XVI).

Sources

One of a series of field notes on building software for the way minds actually work: tired, distractible, ordinary, and now partly machine. They all lead back to the manifesto behind them, The Shape of the System.