Regression to the mean

I’ve watched it happen more times than I’d like to admit. A capable data science team ships a strong first model, fast. Then the data drifts, or the business reframes the question, or the one person who really understood the pipeline leaves — and the whole thing seizes up. The model was fine. The people were good. So what breaks?

I used to blame the usual suspects: thin documentation, undocumented features, a notebook nobody else could run. None of those explanations survived contact with reality. The teams that adapted didn’t have tidier repos. The teams that fell apart when someone left often had extensive handovers. Something else was going on.

The answer I keep coming back to is this: the most valuable thing a data or AI team builds isn’t its code or its models. It’s a shared, evolving theory of the problem. And we have just handed every team the most capable assistant in history. One that writes the code, drafts the analysis, and proposes the architecture in seconds without asking whether it helps build that theory or quietly erodes it.

It does some of both. For those of us who build AI for a living, sorting out which is which has become the most important judgment we make.

In 1985, Peter Naur argued, in Programming as Theory Building, that the essence of programming isn’t producing code — it’s building a theory: a deep understanding of how messy real-world activity maps onto a formal solution. I think the idea applies even more forcefully to data science than to the software engineering Naur had in mind. In ordinary software, the requirements are often comparatively fixed; you’re told what to build. In data work, the problem itself is underspecified. You’re deciding what question the data can even answer, what the target variable actually represents, which metric reflects the business reality, and which merely flatters it. The modeling and the math are the teachable, explicit part. The theory is the larger, mostly invisible part.

The model is the residue of that theory, not the theory itself. Which is exactly why a perfectly reproducible notebook can become unmaintainable the moment its author leaves. The repo captured the features, the loss function, and the final metric. It never captured why those features and not others, what was tried and rejected, what the team came to believe the data really meant.

Here’s the uncomfortable part. Much of what makes a senior data scientist effective can’t be written down at all. Michael Polanyi put it simply: we know more than we can tell. This tacit knowledge is everywhere in our work — the intuition that a result is suspiciously good and probably leaking, the smell of an overfit before the validation curve confirms it, the sense that a feature is quietly a proxy for the label, the feel for when a model is good enough for the decision it will actually inform.

This is why two teams with identical CVs and the same tooling produce wildly different outcomes. The best ones find ways to share and grow their tacit knowledge, usually through close collaboration rather than a formal process. You can’t memo your way to it, and, as we’ll see, you can’t prompt your way to it either.

If so much of a team’s expertise is tacit, it follows that there’s rarely a single correct solution. Philosophy of science has a name for the underlying principle: the Quine-Duhem thesis, which holds that data underdetermine theory. The same observations can be explained by multiple, equally valid frameworks. The parallel to modeling is almost too on-the-nose: many architectures fit the same data, many feature sets hit the same metric, and many framings of the problem are defensible. The data never picks the model for you.

So when your team commits to one path among several valid ones, you’re not just making a technical call. You’re encoding values and priorities — what kind of error is tolerable and for whom, what “good enough” means, what you’re really optimizing. The choice reveals who you are as a team, which is why it matters enormously who and what is in the room.

My hardest-won lesson about building teams: homophily, our pull toward people who think like us, is both powerful and dangerous. A team of people with similar training and mental models is a joy to run. They finish each other’s sentences. Shared assumptions don’t need validating, so they move fast. And they share identical blind spots. Everyone reaches for the same architecture, frames the problem the same way, and no one is wired to challenge the assumption that needs challenging. The team’s theory fits its own worldview beautifully while quietly missing a chunk of reality.

I learned this the expensive way. I once worked on an enterprise search problem with a brilliant, like-minded group. We squeezed everything we could out of Solr and its ecosystem. And we completely missed that traffic was shifting hard to mobile. Turned out that what we’d framed as a sophisticated relevance-and-ranking problem was really an information-architecture problem that wanted a simple predictive-input solution. We had hammers, so everything looked like a nail.

The contrast came later. When a startup of ours needed to pivot, a team of six (four technical, all from different backgrounds) went from idea to working proofs of concept in weeks. The cognitive diversity meant we could see multiple framings the moment we needed them. Mixing ML people with domain experts, social scientists, physicists — people who decompose a problem differently — isn’t just enriching. The friction is the mechanism. It’s how a more robust theory gets built.

Now put a large language model on that team. I want to be clear up front: I love these tools and use them every day. But we should be honest about what kind of colleague we’ve added.

Structurally, an LLM is the ultimate homophilous teammate. It’s trained to emit the most probable continuation; the modal, consensus answer. It literally finishes your sentences. So it doesn’t merely fail to add diversity; it pulls toward the mean, and it does so by collapsing exactly the Quine-Duhem space we just celebrated. Ask it how to approach a problem, and it will surface the single most common approach, fluently and confidently, hiding the fact that a choice was ever open...

Multiply that across a field where everyone leans on the same few models, and you get correlated blind spots at scale. The upshot inverts my original lesson: human cognitive diversity is no longer just an edge. It’s the counterweight to a homogenizing force we’ve invited into every team. It is the variance the tool can’t supply and actively dilutes. And there’s a particular irony for us: the system we’re tempted to hand our judgment to is the very kind of system we build.

It would be easy to stop there and sound the alarm. That would be wrong, and you shouldn’t trust anyone who does it.

Andy Clark and David Chalmers argued decades ago that the mind doesn’t stop at the skull. When a tool becomes reliably available and seamlessly used, it becomes part of your cognitive process — a notebook becomes part of your memory, a phone becomes how you remember numbers. Offloading degrades the unaided ability but usually buys a more powerful extended one. Net positive. By this logic, extending a team’s mind with AI is just the next chapter of a very old story, and most of the panic about it is misplaced. Clark himself is optimistic that generative AI fits this benign pattern.

I think he’s mostly right. That is exactly why the one place he’s wrong matters so much.

Duncan Pritchard draws a distinction that, once you see it, you can’t unsee. Most cognitive abilities are specific, like recalling a number or identifying an aircraft. Offload those to a tool, and you’re fine; the extended version is better. But a second class he calls executive cognitive abilities, e.g., being observant, being intellectually conscientious, exercising critical judgment, behaving differently. They aren’t innate; you acquire and keep them only through practice. They have broad reach. And they play a managerial role: they govern how all your other abilities get used. You can use a tool to aid them, but you can’t hand them over.

Pritchard’s worry is that the more capable and autonomous AI becomes, the more it absorbs the very activities through which these executive abilities are grown and maintained. How do you sharpen critical judgment if you never work the problem yourself? How do you build modeling intuition if the tool proposes every model? The loss is different in kind from losing mental arithmetic to a calculator, and it isn’t offset by the impressive new extended abilities. Without good executive judgment, those abilities get deployed less wisely.

Here’s where it lands for data teams. Naur’s theory-building is an executive activity. Judging which framing of a problem is right, sensing when a model no longer fits the world, knowing which approach to reach for, spotting the assumption that has to be questioned. That’s the managerial, metacognitive layer Pritchard is describing, operating at team scale. A team’s theory is its collective executive capability. It is the one thing you cannot extend into the model without hollowing out the team.

So treat the LLM as what it is: a genuine member of the team that does the specific work superbly — boilerplate, scaffolding an analysis, a first pass at features, looking up a method, the tireless production of artifacts. Extend your team’s mind with it freely there.

But it cannot be the custodian of the theory, because theory-holding is executive and non-extendable. The model has no persistence of your team’s evolving understanding of the problem, no stake in the outcome, no accountability, and none of the tacit, hard-won judgment the theory is made of. Let it quietly become the place your theory lives, and you’ve recreated the failure I started with — the understanding walking out the door — except now it was never really held by anyone. The drift comes, the metric stops meaning what you thought, and there’s no one home to notice.

None of this means keeping the tools at arm’s length. It means using them on purpose. Pritchard separates active integration — deliberately adopting a tool and managing how it fits your thinking — from passive drift, where a tool is so capable and ready-to-hand that it takes over your judgment before you notice. Capable AI invites the drift. Resisting it is a leadership practice, not a personality trait.

In practice, a few habits:

• Frame the problem before you prompt. The framing is the theory-building. Outsource it, and you’ve outsourced the part that mattered.

• Judge the output; don’t adopt it. Treat AI work as a strong proposal to be interrogated, not an answer to be accepted — the way you’d treat a confident colleague who is sometimes brilliantly wrong.

• Keep doing some of it by hand. We all use calculators, but a careful analyst still reruns a suspicious figure with pen and paper — not because the calculator fails, but to keep alive the judgment that knows when a number is off. Read the errors yourself. Build the mental model of the data yourself. Sometimes, frame a problem on a whiteboard with no model in the room.

• Defend the diversity that the model erodes. It’s now load-bearing, not decorative.

This reframes the ordinary rituals, too. Code and model reviews become a check that humans still understand the AI-written pipeline, not merely that it runs. Documentation has to trigger genuine reconstruction in someone’s head, not sit unread because the model can re-explain on demand. Cross-training is about keeping the theory in human custody, not just covering the bus factor.

Building data and AI teams has always meant balancing immediate delivery against long-term resilience. AI shifts both sides of that trade at once: it makes the specific work nearly free, and it makes the executive work — the theory-building — far easier to skip without noticing it’s gone.

The strongest teams won’t be the ones that adopt AI fastest or resist it hardest. They’ll be the ones that decide, deliberately, which thinking stays human. Offload the specific. Protect the executive. Keep the theory in human custody, on purpose.

The best teams don’t just ship models. They build theories of the problem that survive drift, departures, and pivots. In this era, that means knowing exactly which part of the work they refuse to hand away. The sharpest irony belongs to our own field: the people building AI are the most exposed to offloading the very judgment that makes them good at it.

The question was never how well your team models. It’s how well they think together and whether they’ll still know how, once the machine offers to do it for them.