The Error He Couldn’t See

He had sent it over the night before, and it was clean. The formatting was right, the structure was sound, and the summary at the top read like something a senior analyst would write. I was three lines into the section that carried the most weight when I felt the thing I have learned to trust more than almost anything I can defend out loud. A gut instinct that something was off. The sense that something crucial was missing under one of the numbers.

I asked him where the figure had come from. He told me, with complete confidence and no hesitation, and that is the part that stayed with me for weeks. He was not bluffing. He genuinely could not see what I could see. The work looked finished to him because, from his view, it was.

The gap between us in that moment had nothing to do with intelligence. He is sharper than I was at his age, and faster than I have ever been. What I had and he did not was a quiet alarm, a sense that something was off before I could explain why. I have started to wonder where that alarm came from.

The honest answer is that mine came from years of doing the work badly.

Long before I could trust my own read on a number, I produced a great deal of confident, tidy, wrong work of my own, and I had people above me who caught it, sent it back, and made me sit with what I had missed. The feedback was the education. Each correction laid down a little more of the pattern, until one day I noticed I was flinching at things before anyone told me to.

What he was missing is older than any of this. There is a line from the philosopher Michael Polanyi I keep coming back to. We know more than we can tell. The most important things a skilled person knows are the things they could not write into a procedure, the feel for when an answer is wrong in a way that looks right, the instinct that a clean number has a rotten assumption underneath it. Polanyi’s point was that this kind of knowing moves almost entirely through apprenticeship. You watch someone who has it, you try it yourself, you get it wrong, and slowly the tacit thing transfers without either of you being able to name what just passed between you.

The clearest picture I have found of how that works comes from research on firefighters. Gary Klein spent years studying fireground commanders who could walk into a burning building and know, without running through options, that the floor was about to give. When he looked at how they decided, he found they were not weighing alternatives at all. They were recognizing a situation they had effectively seen before. The commanders he interviewed had spent an average of twenty-three years on the job. The intuition that looked like magic was pattern recognition, learned one fire at a time.

Klein eventually found unlikely agreement with Daniel Kahneman, of all people, on the conditions that make that kind of intuition trustworthy. The world has to be regular enough to send the same signal twice, and the person has to get feedback fast enough and clear enough to learn the signal. A messy environment with slow or muddy feedback will not build the alarm, no matter how many years go by. The young man on my screen had the talent. What he had not yet been given was the second condition, the long run of fast corrections that turns talent into a sense.

And that long run of corrections is exactly the part we have started handing to the machine.

There is a reason the missing alarm is not a matter of effort. It sits in how the brain actually works.

For about a decade, the researchers who study perception have converged on a strange and useful idea, that the brain runs on prediction. The philosopher Andy Clark and others describe our experience of the world as a controlled hallucination, the brain’s best guess about what is out there, checked against what comes in. On this view you do not really notice the world directly. You notice the difference between the world and what you predicted. You register something as wrong only when it breaks a prediction you were already holding.

This is the whole thing, sitting under that quiet exchange about a number. My alarm went off because that figure did not match a prediction twenty years of being wrong had taught me to make. He had no prediction for it to break. He was not being careless or lazy. He had nothing to compare the answer to, so he read it as true. You cannot catch the error you have no prior experience with.

I am not the only one watching this from the other side of the screen. Researchers at Microsoft and Carnegie Mellon surveyed knowledge workers using generative AI last year. The ones who trusted the tool blindly were the ones who thought the least about what it produced. The people who felt most sure about the machine did the least of the checking that would have caught it being wrong. The ones most exposed are the people who had not built the prior in the first place, paired with a tool that produces its most confident output exactly where the most doubt is warranted.

What unsettles me most is that we were warned, long before anyone could have meant it about AI. Back in 1983, an engineer named Lisanne Bainbridge wrote a short paper called the Ironies of Automation that has only grown more accurate with age. Her warning was that when you automate the easy parts of a job, you leave the human with the hard parts, while quietly removing the daily practice that made the hard parts possible. The operator becomes a monitor of a system they are slowly losing the ability to question. She was writing about power plants and cockpits. We have now reproduced that same irony in the center of knowledge work, with one cruel addition. The easy part we automated was the apprenticeship itself.

The first drafts, the rote analysis, the boring reconciliation work, the thing you used to hand the new person precisely because it was low stakes and high repetition. That was more than grunt work. It was the factory floor where judgment got manufactured, one corrected mistake at a time. We looked at that floor, saw how slow and inefficient it was, and handed it to a machine that produces the same result in seconds. The corrected mistakes that used to shape a person stopped happening.

You can already see the rung being pulled up. Stanford’s Digital Economy Lab, in a study Erik Brynjolfsson and his colleagues pointedly titled Canaries in the Coal Mine, found a thirteen percent relative decline in employment for early-career workers in the most AI-exposed jobs since generative tools arrived, while employment for older workers in the same roles held steady or grew. Young software developers were down close to twenty percent from their late-2022 peak. The numbers are still moving the same direction this year. A generation is being asked to form a working identity without the years that used to build one.

We automated more than the tasks the juniors used to do. We automated the apprenticeship that turned them into the people who could tell when the work was wrong.

The young man will be very good one day. I believe that. But he will only get there if someone deliberately protects the conditions that built the alarm in me, the chance to do real work, get it wrong, and be caught by a person who can still see what he cannot yet. Those conditions used to be automatic. They were the natural shape of starting out at an entry level. Now they are something a few of us will have to choose to preserve on purpose, against every incentive to ship the finished-looking output and move on.

What I keep circling, in the quiet after that call, is a simple and uncomfortable question. When the people who built their priors the old way retire, and the people coming up behind them were handed a machine instead of an apprenticeship, who is left who can raise the alarm when something is missing or just wrong?

We are very good right now at producing work that looks done. What we are quietly losing is the sense for the difference between work that looks done and work that is right, and that difference is the work.

I notice I am no longer worried about the machine being confidently wrong. It will be, often, and that is almost manageable. What I am sitting with is the picture I keep returning to a decade from now. A room of capable people who came up in this new hybrid AI workforce without the chance to be wrong on purpose, looking at an answer that looks flawless, and feeling nothing at all.

Watching the gap,

Yen