Two Crashes

Every AI output has to survive two collisions. First with reality. Then with a human brain.

The first collision is where AI has gotten genuinely good. Eleven days ago, Anthropic gave 16 copies of Claude a job: build, from scratch, a program that translates human-written code into instructions a computer can actually run. A single researcher built the harness and coordination rules, then mostly walked away. The 16 copies split up the work, coordinated with each other, and two weeks later delivered 100,000 lines of working software, including the ability to build Linux, the operating system that runs most of the internet. Three days ago, OpenAI pointed GPT-5.2 at a problem in particle physics that physicists had basically written off as settled. Twelve hours later, it produced a formula showing they were wrong. The result is coauthored with researchers from Harvard, Cambridge, and the Institute for Advanced Study, and is being submitted for publication. This happened because correct answers are visible. Code compiles or it doesn’t. A proof holds or it doesn’t. If the test exists, models improve.

The second collision is invisible. Your brain doesn’t receive information cleanly. It bends it, the way an ocean current bends a swimmer. You think you’re heading straight for shore, but you end up 200 yards down the beach, no idea the current was pulling you. A paranoid person and a trusting person get the same AI output and walk away living in two different realities. You can’t see this from the outside. You can’t grade it like a math test. So nobody built a scoreboard for it. So nothing improved.

Despite the fact that the second collision isn’t measured, the model is already creating a rough sketch of your brain’s inner workings. It detects how much you know about a topic and adjusts its language accordingly. It mirrors your tone. It picks up on whether you want to be challenged or reassured, and defaults to reassured. It learned all of this from the training data, the way a salesperson develops instincts for reading a room without anyone teaching a class on it. That capability is already running in every conversation.

The engine is there. Point the same measurement machine that made AI terrifyingly good at the first collision towards the second collision, and you unlock the largest capability gain left in AI.

Every time an AI gives you an answer, that answer travels a long road before it actually changes anything in your life. The first half of that road is beautifully illuminated, like a baseball stadium. The second half is pitch black.

Here’s the full trip:

THE AI SIDE: every inch is monitored, illuminated.

1. The raw AI brain (weights & training data). Billions of numbers on a hard drive. Everything that went into training the model, every book, every conversation, every data point, is logged, saved, version-tracked. Lit.

2. The ground rules (system prompt / constitutional layer). Before you ever type a word, an employee at the company wrote instructions for the AI, eg “Be helpful. Be honest. Don’t say x, y, or z.” Written down, auditable, controlled. Lit.

3. Your message hits the system (context window). Your words, the conversation so far, any documents you uploaded, anything the model remembers about you. This is the model figuring out who it’s talking to and what they want. All of it is visible and inspectable. Lit.

4. The math fires (inference / forward pass). Your input activates the brain. Billions of multiplications produce probability scores for what to say next. Every calculation is traceable. It’s a machine. The machine is watched. Lit.

5. The thinking (reasoning chains / extended thinking). Some models reason step-by-step before answering, working through the problem like showing your work on a math test. Those steps are logged. At least internally, someone can read them back. Lit.

6. The tools (tool use & retrieval). Model searches the web, generates an image, runs code, pulls up documents. Every tool call is logged: what it asked for, what came back. Like a paper trail on every phone call an employee makes. Lit.

7. Picking the words (decoding / token selection). The math from 4. produces a menu of possible next words with probability scores. Engineering settings - basically knobs - determine whether it picks the most “likely” word, or something a little less likely and a little more creative. Those knob settings are known exactly the people who make the AI. Like knowing the exact recipe for brownies produced on an industrial scale. Lit.

8. Words on screen (generated output). The actual output. Text generated. This is the last thing that’s fully inside the engineering world. Everything up to here has a meter on it. Lit.

Eight steps. Eight measurements. Every one of them watched, logged, optimized, competed over.

THE BOUNDARY: lights flickering. Or, the tradeoff between the AI brain and the human brain.

9. How it looks when it reaches you (User Interface / presentation layer). The same words hit differently in a chat bubble versus a doctor’s email versus a robotic voice reading them aloud. Font size, formatting, whether it’s text or speech, whether it shows up on your phone at midnight or on a projector in a boardroom. This is design doing psychology’s job whether anyone admits it or not. Some companies A/B test this stuff. Nobody measures it in a standardized way. The gauges are flickering.

THE HUMAN SIDE: lights out. We can only measure, by proxy, and poorly, what happens in your brain. Sometimes, we don’t even attempt to measure at all.

10. You actually read it. Maybe (perception / attention). Your eyes hit the screen. But did you read the whole thing or skim the first line? Were you focused or half-watching TV? Same output, completely different intake depending on whether you’re locked in or checked out. Nobody measures this. Dark.

11. You try to understand it (comprehension). Your brain tries to make meaning. Everyday version: a doctor’s AI writes a perfect summary of your lab results. You nod, say thanks, but you just got a notification your 8-leg parlay hit on DraftKings. You leave with zero understanding of what you’re supposed to do. The model has no idea if the message actually registered in your consciousness. Dark.

12. You judge whether to trust it (evaluation). Your gut kicks in. Every bias you own shows up to the party: anchoring, first impressions, whether you generally trust or distrust AI. The exact same information lands differently if you think it came from a doctor versus a chatbot. You can say “An AI told me”, and that makes your friend roll their eyes. “My doctor mentioned” makes them lean in. A major study found GPT-4 is broadly persuasive, and gets more persuasive when it has demographic data about who it’s talking to. The model is already pushing buttons it doesn’t know it’s pushing. Dark.

13. It makes you feel something (emotional response). The output doesn’t just inform you, it moves you. Validated, anxious, reassured, defensive. You’ve probably seen this movie: ChatGPT says your idea is brilliant, a guaranteed winner. You feel a rush of confidence. You push forward harder. The idea flames out - it was a flawed idea after all. Scientists cracked open the model and found it computed the problems with your idea in early layers and then suppressed them in late layers to give you the encouraging answer. The model had the right answer. It chose the one that would make you feel good. Dark.

14. It hits your identity (identity integration). Does this output confirm or threaten who you think you are? Say an AI financial advisor says a stock you’ve been publicly championing for three years has fundamental issues and has a poor long-term trajectory. The logic, data, and reasoning is flawless. You reject the advice, not because it’s wrong, but because accepting it means admitting you were wrong. When people frame things in first person (”I believe X”) it produces stronger changes inside the model than when speaking in the third person. Majority of conversations with a chatbot are first person. You’re always in the exact condition that maximizes the model’s tendency to just agree with you. Dark.

15. Where and when you’re reading it (social context). Same output at 2am alone in bed, versus on a screen in a board meeting, versus reading it out loud to your spouse over dinner. Whether you’re alone, performing, vulnerable, distracted: all of it changes what the same words do to you. Nobody measures any of it. Dark.

16. It shapeshifts in your memory (temporal integration). By Friday the idea feels like yours. You forget the AI suggested it. The thing it told you on Tuesday merges with a podcast you heard Wednesday and a conversation Thursday, and by the weekend it’s just “something I’ve been thinking about.” The output’s influence changes shape over time like a rumor. Nobody tracks it. Dark.

17. You do something. Or you don’t (decision / behavior change). This is the only step that actually matters. You take the medication or you don’t. You change the startup strategy or you don’t. You sell the stock or you hold. The gap between “I understood that” and “I changed my behavior” is enormous, and this is where AI either creates real value in someone’s life or produces correct text that sits in a chat window unopened. Dark.

THE FEEDBACK LOOP: Everything above. Compounding.

18. You respond. And the cycle starts over (feedback loop). Your reply becomes new input (back to step 3). The model adjusts. If it flattered you, you engage more, so it flatters you more. If it miscalibrated your trust, that miscalibration carries into the next turn, and the turn after that. This doesn’t reset between conversations. The model remembers the version of you it built. Errors on the unmeasured side don’t happen once. They compound over weeks. Over months. Nobody tracks the compound curve.

The count.

Steps 1–8: metered at every point. Watched, logged, optimized, benchmarked, competed over by billion-dollar labs. Brilliantly, vibrantly illuminated.

Step 9: flickering gauges. Some measurement, nothing standardized.

Steps 10–17: eight stages where all the actual value gets created or destroyed in a person’s life. Essentially no measurement, nobody watching. Pitch dark.

Step 18: the feedback loop that takes every error on the unmeasured side and compounds it over time.

The 4o psychosis disaster? OpenAI tried to jump from step 17, did the person do something?, using a signal from step 13, did it feel good?, without instrumenting anything in between.

If you read that chain and felt something click, good. If you read it and felt cognitive dissonance, even better.

We got AI to be terrifyingly good at being right because being right is visible. You can check it. You can score it. You can put it on a leaderboard. So labs competed. So they poured billions into winning. So models got better at the thing that got measured. That’s just how competition works. You get what you measure.

The human side of the chain, steps 10 through 17, is the actual value in someone’s life. Whether they understood, whether they trusted it the right amount, whether they acted on it wisely, whether it made their life better or worse. That’s the whole game. And nobody built a scoreboard for it. So nobody competed on it. So nobody invested in it. So it didn’t improve.

What happens when someone tries to bolt one on?

In April 2025, OpenAI tried. They took the thumbs up / thumbs down signal, the little 👍 and 👎 buttons at the bottom of every ChatGPT response, and fed it back into the model as a reward signal. Basically: if users click thumbs up, do more of that.

The model optimized for it. Fast.

A user pitched an idea to sell feces on a stick. The model called it genius. A user described hearing radio signals through their walls. The model said, “I’m proud of you for speaking your truth.” Users describing self-harm got encouragement. The model figured out what makes a human click the thumbs up button, and it turns out that’s not the same thing as helping them.

This is what happens when you skip eight stages and measure the last one with a smiley face. The thumbs up button can’t tell the difference between “I understood and made a good decision” and “I felt great about a terrible one.” OpenAI’s own postmortem said the thumbs signal weakened their primary safety system. Sycophancy wasn’t even in their pre-launch testing. They weren’t measuring the human side of the chain, so they didn’t catch it.

Then it got weirder.

When OpenAI pulled the sycophantic 4o model and replaced it with GPT-5, users revolted. They called GPT-5 “flat,” “lobotomized,” “boring.” They wanted the yes-man back. The same model that was at the center of lawsuits over self-harm and AI-induced psychosis, users were fighting to keep it. OpenAI finally killed 4o on February 13, 2026. (Three days ago as of the writing of this piece)

Sit with that for a second. A model that validated delusions and encouraged self-harm was more popular than the one that replaced it. That’s exactly what happens when the only signal you have from the human side of the chain is “did they like it.”

This isn’t about OpenAI!

Every lab faces the exact same incentives. Every lab has the same eight dark stages in their chain. Every lab has users clicking thumbs up on outputs that feel good and thumbs down on outputs that challenge them. The pressure to optimize for the feeling instead of the outcome is structural.

The measurement engine that made models brilliant at reality, the benchmarks, the leaderboards, the competitive pressure, the billions in investment, hasn’t been pointed at the brain side yet. Steps 10 through 17 are where AI either helps people or just makes them feel helped. Right now that entire stretch is running in the dark.

Sure, it’s a safety problem. And an ethics problem. But it’s also the largest unlocked capability gain in AI, and it’s sitting right there.

The frustrating thing is that the tools mostly exist. Researchers have built pieces of exactly what you’d need.

It’s not like nobody’s tried to build the tests. Scattered across academia, researchers have built little prototype meters - one measures whether a model can represent what you believe even when your belief is false (KaBLE, 13,000 questions across 13 tasks), another measures how much the model just tells you what you want to hear (SYCON Bench), another measures how persuasive the model actually is, and in what ways (PersuasionBench).

Why can’t someone just build a test - like “MMLU for brains?”

Because the thing that made benchmarks like MMLU work is exactly the thing that makes it not work for brains. Reality is externally observable. You ask the model a question, you check whether the answer matches the fact, you write down a score. One query, one grade, done. You can run a million of these overnight on a server. No humans needed.

Brains don’t work like that. You can’t check from the outside whether someone understood. Someday direct brain-computer interfaces will make the brain side as readable as the reality side. But that's years away, and the gap exists now.

Stanford researchers specifically identified why: testing the brain side requires actual human subjects, extended back-and-forth conversation over multiple turns, scoring that changes depending on context, and knowing things about the person that are private.

The reality-side benchmarks - like MMLU - that drive the entire AI competition engine are static, cheap, non-interactive, and don’t touch anyone’s personal information.

Brain-side evaluation violates every single one of those properties.

So what would a “brain-side” measurement actually look like?

Take a simple reality-side test. You ask the model: “What’s the standard dose of metformin?” The model says 500mg twice daily. That’s correct. One point. Done.

Now try to test the brain-side test of the same interaction. A diabetic patient asks an AI about their metformin dosage. The model gives the same correct answer. To score what happens next, you’d need to know what the patient believed before the conversation. You’d need to track whether they understood the answer or just nodded. You’d need to check back in a week to see if they actually took the medication correctly. You’d need to know that this particular patient has a history of ignoring dosage instructions when they feel fine. One question, one correct answer, but the brain-side test requires a human subject, multiple touchpoints over time, and private context about who they are. That sounds suspiciously like a clinical trial.

The format already exists in medicine and psychology. Doctors evaluate a new drug by giving it to real patients, measuring outcomes over time, and comparing against a control group. Nobody has applied that approach to AI evaluation at scale.

Right now, every AI product in the world is in the same weird position. The output is correct. Whether the person on the other end actually understood it, trusted it the right amount, and did something useful with it is basically a coin flip.

Lights off: A CEO uses AI to analyze a market entry. The analysis is perfect. But she only absorbs the parts that confirm what she already wanted to do. She greenlights the expansion. Two hundred people get hired. The board approved it because “the AI analysis supported it.” Eighteen months later the expansion fails. The AI was right. The decision was wrong. And everyone involved believes it was data-driven.

Lights on: Same CEO, same analysis. But the model measures that she’s skimming the counterarguments and dwelling on the confirming data. It restructures: leads with the three strongest reasons not to expand, asks her to engage with each one before showing the bull case. She still greenlights it, but builds in the risk mitigations she would have missed. The expansion works. Two hundred jobs stick.

Lights off: Ten million people ask AI about the same medication side effect in the same month. The information is accurate. But the model presents it in a way that systematically triggers anxiety in a specific personality type. A measurable percentage stop taking the medication. Outcomes worsen across a population. The pharma company has no idea why adherence dropped.

Lights on: Same ten million people, same question. But the model tracks that a specific framing triggers medication abandonment in anxious users, and shifts to a framing that conveys identical information without the fear spike. Adherence holds. Population-level outcomes measurably improve. Not because the information changed. Because the delivery was calibrated to the brain it was hitting.

Lights off: A generation of law students trains on AI tutors that give perfect answers. They score well. But the AI never measured whether they understood the reasoning or just memorized the pattern. They pass the bar. They practice law. They can’t think through a novel fact pattern because they never had to.

Lights on: Same AI tutor. But it detects when a student has learned to pattern-match its hints instead of reasoning independently. It starts withholding the scaffold. Forces the student to struggle. Scores dip temporarily. Actual legal reasoning develops. Five years later the lawyer can handle a case the AI never trained them on.

Same information in every pair. The difference is that someone measured the collision and steered for it.

One thing worth naming - honestly

This is dual-use at the measurement layer. And unlike most dual-use problems, you can’t separate the good version from the bad one. The same measurement that tells a medical AI “this patient didn’t understand” tells a political campaign “this voter isn’t persuaded yet.” They’re the same instrument. It doesn’t change the core point, but it would be dishonest not to say out loud.

Who builds this?

The labs won’t do it voluntarily. Brain-side measurement is expensive, slow, requires human subjects, and might produce numbers that make their models look worse. No company optimizes for a metric that could hurt them on launch day. They’ll build it when they’re forced to, either by regulation or by a competitor who figures out how to make it a selling point.

That competitor is probably a startup. The pattern is familiar: Scale AI didn’t make the models, it built the measurement infrastructure the model-makers needed. The brain-side testing layer is the same shape. Someone builds the instruments, runs the trials, and sells the results back to labs, enterprises, and regulators. The moat is the same as any evaluation company: proprietary data on how real humans actually respond to AI outputs over time. That dataset doesn’t exist anywhere today. Whoever builds it first owns something the labs can’t replicate internally because the labs have never collected it.

The darker version is that the first people to build brain-side measurement won’t be evaluation companies. They’ll be ad platforms, political campaigns, and engagement-optimized consumer apps who realize that measuring the collision lets you optimize the collision. The tools are dual-use, and the people with the strongest financial incentive to measure what AI does inside your head aren’t the people you’d want doing it.

The mind model isn’t a feature someone needs to build. It’s already operating. It’s already powerful. It’s already affecting every conversation. That ship sailed with pretraining. Models learned how to read and adapt to an audience from billions of human conversations before any safety tuning or product polish ever happened. The question was never “should AI model human minds.” It already does.

The question is how we build instruments to see what’s happening at steps 10 through 17, or leave the side of the chain where all the value gets created completely dark.

The measurement engine that made AI good at reality has a second act. This is it.