The Relocation of Fragility : Innovation doesn't eliminate fragility. It redistributes it.

I remember the meeting where it started. End-of-month review. Big projects, the company I worked for at the time. Someone announced the multiplex experiment (twelve markers, twelve colors) would be ready in a month. So I prepared. Weeks of it. Protocols, new analysis frameworks, papers on twelve-channel imaging. I wanted to walk in ready the day the data showed up.

It didn’t. One channel, too much noise. Another, nothing. Crosstalk between markers that were never supposed to touch each other. Someone would fix a problem and another one would surface, different channel, different sample, same headache. My analysis protocol sat in a folder. Waiting. For three months.

The experiment that should have taken a month took four.

With a single marker and a single color, the workflow is almost boringly simple: the pathologist looks at the sample, and if the stain came out a little pale, reads it anyway. Notes it, moves on, adjusts the protocol next round. The system tolerates error and keeps working. Twelve colors don’t work that way. Each new channel doesn’t add to the last one, it’s one more point where the whole experiment can break.

For months I chalked it up to bad luck, or a project that was just unusually hard. It wasn’t either. It was something plainer, and honestly more uncomfortable to sit with: every innovation promises to remove a problem. Almost none actually does. What it does, nearly every time, is move the problem somewhere else.

What I lived through in that lab wasn’t new. Someone had already described it, decades earlier, looking at a completely different problem.

In 1983, Lisanne Bainbridge wrote a short essay on industrial automation that today reads like prophecy. She wasn’t writing about hospitals or labs. She was writing about control rooms, operators in front of factory panels. But she described, forty years ahead of time, the exact mechanism now playing out in precision medicine.

Her argument breaks into two parts. Both matter, equally. First: automate an operator’s routine tasks, and you don’t make that operator unnecessary. You leave them, precisely, the hardest tasks, the ones nobody figured out how to automate. The ones that demand the most judgment, right when the operator has the least practice at exercising it.

The second part is the one that gets under my skin. Pull an operator out of daily control, and you also pull away the constant practice they needed to intervene well the day something breaks. We ask them to passively watch a system that almost never fails, and then react with expert precision at exactly the moment, rare and unexpected, that it does.

Take a pathologist who now reads “positive” or “negative” off a screen. Slide after slide, they’re no longer building the instinct that used to let them clock a strange case on sight. So when the algorithm fails for a reason that has nothing to do with biology (a bad staining batch, say, or a different scanner) that pathologist simply has less recent practice to catch it. Bainbridge never needed to see a single AI model to see this coming.

A year later, Charles Perrow published Normal Accidents, after investigating the Three Mile Island nuclear accident. Bainbridge explains what happens to the human operator. Perrow explains what happens to the system.

Two ideas do most of the work here. Interactive complexity: so many components, tangled up with each other in ways nobody designed on purpose, that one tiny failure combines unpredictably with another tiny failure somewhere else entirely. And tight coupling: rigid sequences, zero slack between one step and the next, no time for a human to catch an error before it’s already spreading.

Put both properties in the same system, Perrow argues, and serious accidents stop being some remote possibility you engineer away. They become “normal.” A property of the system. Not a failure of execution.

And there’s something even more uncomfortable: adding safety layers doesn’t always reduce risk. It often increases it. Every new layer brings its own interface, its own failure mode, and nobody fully understands how they interact.

Bainbridge and Perrow aren’t describing two problems. They’re describing the same phenomenon from two different altitudes.

The same move that weakens the human who might catch the failure is, almost always, the same move that makes the failure propagate faster than any human could catch it.

That’s not a coincidence. Both come from the same decision: chase consistency without asking who’s going to absorb the variation that keeps existing, always.

Stripped to its skeleton, the process is always the same:

This sequence doesn’t mention artificial intelligence. It doesn’t mention pathology. It works the same way in an airline, a bank, a nuclear plant. AI didn’t invent this rule. It just makes it visible much faster, compressing into months what took other industries decades.

Variability used to have exactly one place to resolve: the pathologist’s judgment. Today it’s spread across six or seven different links, and none of them absorbs error with the ease a trained eye once did.

Adaptation no longer lives in one person. It’s distributed, and, for the most part, invisible.

I come back to my own territory now. I’m not choosing digital pathology because it’s a special case. I’m choosing it because, of all the industries where this mechanism operates, it’s one we can watch most clearly, with the freshest evidence.

The most revealing part isn’t that models fail obviously, in public, in ways anyone would notice. It’s what they’re learning underneath, while everything looks like it’s working.

The system wasn’t learning biology. It was learning provenance. It’s Bainbridge again: the model became an expert at the easy task, and stayed blind to the hard one nobody taught it well, because nobody knew how.

Once that kind of algorithm becomes the core of a biomarker selecting patients for a Phase III trial, it stops being a technical curiosity.

A pathologist degrades gradually. Worth pausing on that, because I don’t want it to sound like idealization: pathologists are people, not machines. Nobody delivers excellent performance every single day, across an entire career. What sets their kind of failure apart isn’t that they don’t fail. It’s that they don’t fail all at once.

Faced with an imperfect sample, a pathologist doesn’t just stop working. Doubt shows up first. Confidence drops, a notch at a time. They start asking for a second opinion more than they used to. And almost always, they know it’s happening, they know when they’re seeing worse than they did a few years back, know when the mistakes are creeping up, and more often than not, can tell you themselves when their sharpest years have already passed.

A model doesn’t degrade that way. It holds near-perfect performance right up to the exact instant a sample falls outside what it learned. There, it doesn’t dip, it collapses, with the same numeric confidence it had when it was right. No prior doubt. No threshold sliding slowly. Nobody inside noticing something’s starting to go wrong.

A pathologist knows when they’re reaching their limit, even when that limit arrives with age, not with a hard sample. A model almost never knows it reached its own.

In 2018 and 2019, Boeing wanted to compete faster with the new Airbus A320neo. It fitted larger, more efficient engines onto the classic 737 airframe. The new engine size changed the plane’s aerodynamics just enough that, in certain maneuvers, the nose tended to pitch up on its own.

Instead of redesigning the airframe (expensive, slow, full recertification required) Boeing solved it with a layer of software: MCAS, which pushed the stabilizer down automatically. The system relied on a single angle-of-attack sensor, with no redundancy. When that sensor failed, the software pushed the nose toward the ground, again and again, overriding the pilots. They didn’t know MCAS existed.

Interactive complexity and tight coupling, exactly as Perrow described them: one invisible component that reduced the slack between a faulty sensor and a tragedy to zero.

The second case is more mundane: software infrastructure. A single server fails simply, it goes down, you bring it back up. The industry moved to distributed systems on Kubernetes precisely to eliminate that fragility.

But at scale, documented industry practice shows the same pattern: the fragility didn’t disappear. It relocated into the configuration across dozens of nodes. Small, undocumented variations (configuration drift) produce failures that are hard to isolate, because no single component is broken. The problem lives in the interaction, not the part.

Nobody in that end-of-month meeting was doing anything wrong. Every decision made sense at the time. That’s the whole point: fragility doesn’t show up because somebody screws up. It shows up because the system gets so consistent, so tightly coupled, that there’s no room left for anyone to be a little bit wrong and just keep going.

AI is going to keep transforming precision medicine. Faster every year. The real challenge of the next decade isn’t building smarter algorithms. It’s building organizations that can absorb the new fragility those algorithms will inevitably introduce, somewhere else in the system, when we’re not looking.

Maybe the real measure of technological maturity isn’t how much we automate, but where we choose to place our next fragility.

References:

1. Bainbridge L. Ironies of Automation. Automatica. 1983;19(6):775-779.

2. Perrow C. Normal Accidents: Living with High-Risk Technologies. Princeton, NJ: Princeton University Press; 1999.

3. de Jong ED, Marcus E, Teuwen J. Current Pathology Foundation Models are unrobust to Medical Center Differences. arXiv preprint. 2025;arXiv:2501.18055v2. See also: Kömen J, Marienwald H, Dippel J, Hense J. Do Histopathological Foundation Models Eliminate Batch Effects? A Comparative Study. Presented at: AIM-FM Workshop, NeurIPS; 2024, independent finding, same pattern.

4. Salgado R, Allen J, Gallas B, et al. Agreement Across AI-Based Digital and Computational Pathology Platforms Scoring HER2 in Breast Cancer: Results from the Friends of Cancer Research Digital PATH Project. Mod Pathol. 2025;38(3):100944.

5. Human Factors and Ergonomics Society. Human Factors & Aviation Safety: Testimony to the United States House of Representatives Hearing on Boeing 737-Max8 Crashes. Washington, DC: HFES; 2019.

6. Industry practice documentation on configuration drift in Kubernetes at scale (platform engineering sources, not peer-reviewed literature).