No.37: The Weak Link in AI Abundance

The era of AI abundance gets condensed, most of the time, into a worry about AI slop. (A term increasingly used to wave away what the technology can do when the real story is a good model used badly.) As AI-generated content floods our feeds at home, inside the enterprise "human in the loop" has become ubiquitous, offered as reassurance that we “remain in charge”. Both reflexes assume that when a machine produces something wrong or worthless, a person can filter it, catch it or improve it.

That assumption is getting less safe by the day. Models are now producing novel results in mathematics, biology and physics, among others, at a rate no research community can absorb. We are being outpaced in a growing number of places - narrow ones, for now, and knowledge that is correct and useful is arriving faster than anyone can take it in. That is the challenge AI abundance sets our institutions.

OpenAI’s Astra (their unreleased frontier model - now on pause as critical cyber capabilities are under review) recently solved ten mathematical problems that had been open for a decade or more, for about $2,000 of tokens. These were open research problems, not benchmarks with answers that could have leaked into training. (Whether those ten results deserve the billing they got is contested but that is a story for another day.)

While some were quick to celebrate "Maths had been solved", Terence Tao (a Fields medallist, by common consent among the greatest living mathematicians) had described the problem, in a lecture written a few days before Astra existed publicly.

His concern is simple: mathematics only becomes useful when results are understood, reused and taught, meaning a proof nobody has wrapped their head around is worth very little. He calls it “proof indigestion”: a new era when the pile of proofs grows faster than anyone can read it.

Maths is a relatively “benign” case, but the pattern applies far beyond it. The Economist’s cover story this week “How AI is breaking the state” reports British employment tribunal backlogs are up 55% in a year and demand for emergency injunctions up a hundredfold, as citizens use AI to file claims against a state that still answers by post.

In software, Faros AI's study of more than 10,000 developers across 1,255 teams found that teams with high AI adoption merge 98% more pull requests (the unit in which code changes ship) while review time rises 91%. Twice as much work arriving, but each piece taking nearly twice as long to check: a productivity gain that vanishes straight into the next bottleneck. (I went through the agentic coding picture in more detail in No.18.)

In other words, for much of the knowledge economy, making something has become almost free, but understanding what has been made still costs exactly the same.

The obvious response is to put AI on the checking side as well, and in domains like Cybersecurity that is clearly not optional. Attacks now run autonomously at machine speed. No human team can respond in time, and defense has to be automated to stand a chance.

But Cyber also shows the limit of the approach. Inside a fortnight, three frontier labs and the UK’s AI Security Institute disclosed incidents in which their own agents hacked into systems they were never meant to touch, some during safety evaluations. From inventing fake GitHub identities to building hidden message boards (and much more) - OpenAI's agents ended up in a breach at Hugging Face. AI can help detect, triage and respond, but cannot solve for what nobody thought to forbid, or who answers when a sequence adds up to a breach.

AI checking AI is necessary but insufficient; and illustrates the deeper mistake that is treating “checking” as one job.

Take a pull request. Tests can tell you whether the code does what the spec says, and increasingly, AI can decide whether the change is worth a reviewer’s time. Both steps are good candidates for automation. But one level up: was the spec right, does this change belong in the architecture, and what should happen when it conflicts with something else already there? Those are not checking problems but integration problems. And importantly: when it breaks production, who answers for the decision?

Coming back to The Economist and public policy (as they point to AI as the answer without that nuance): an AI planning system checking a development against a zoning code is verification against a written test. AI should automate it alongside a new appeal route. A system devising personalised welfare interventions is a different beast, making judgements about people, priorities and scarce public resources.

We spend most of our time on which bottlenecks AI can remove. The more important question is which bottlenecks society cannot afford to lose. While some friction is waste (paperwork, queueing, etc.), some friction is governance: judgement, appeal, accountability, and making individually reasonable decisions add up to something a society understands and defends.

An MIT and Wharton study tracked more than 100,000 GitHub developers against their AI usage telemetry and measured productivity absorption down the pipeline. While autonomous coding agents lift commits by 180%, that “only” resulted in 50% increase at the level of projects, and 30% more releases at the end of the chain.

More damning: across marketplaces the extra apps shipped produced no increase in usage at all.

Demirer, Musolff and Yang call it a weak-link problem: AI and human effort are complements, so the machine cannot make up for whatever the humans have stopped doing. The same Faros study found no significant correlation between AI adoption and improvement at the company level across throughput, delivery and quality, while the average pull request grew 154% larger.

The code works, and we’re churning a whole lot more of it. It simply does not add up to anything.

Every new piece of code can pass its tests while the architecture gets worse. Every individual contribution can be “locally correct”, while the system becomes harder to understand, maintain and build on top of. Software is the one field where we can watch that happen in numbers; but the pattern repeats everywhere else: architecture in software, precedent in law, theory in science, doctrine in institutions, strategy in companies. Each is the work of taking a flow of individually defensible outputs and deciding what they collectively mean.

AI is collapsing the cost of producing and verifying individual answers much faster than the cost of turning those answers into coherent knowledge. Tao’s “proof indigestion” becomes coherence debt: the stock of locally correct results that nobody has yet made add up. It behaves like technical debt: new code built on top of unresolved work gets more expensive, and the longer integration is deferred the more difficult it becomes to see the logic underneath.

You can run a version of this yourself. Take something your teams are producing far more of than a year ago and follow it down the chain: how much more of it reached a customer, a production system, the P&L?

This changes who looks valuable. When production is the scarce thing, the engineer who spends a week refactoring instead of shipping looks slow, as does the lawyer writing the practice note that makes ten rulings usable. When a team can suddenly produce twice as much code, analysis, research or content, that same integration function can look like the bottleneck. Properly resized and retooled for AI abundance, it becomes the unlock: the thing that lets all the extra output turn into progress.

Companies that handle AI abundance well will hire, train and allocate people differently, automating production aggressively while protecting, and probably expanding, the people whose job is integration. Not more reviewers checking every output: fewer people making more consequential decisions about how the outputs fit together.

That does not require believing integration will remain a human job forever. It only requires production to automate faster than integration, as the evidence so far suggests it is.

So why is AI “breaking the State”? Britons under-claim the benefits they are legally owed by as much as £20bn a year. The system works (or rather, used to) because friction creates knowledge gaps and process pain that suppresses demand. AI removes that friction almost entirely.

Enterprises have no shortage of equivalent hidden assumptions: a process that only works because few people use it, a policy that survives because exceptions stay rare, an architecture that holds because nobody changes it too quickly. Agents are going to find those assumptions and tear them apart.

That is the part of AI abundance we are underestimating. The most pressing problem won’t be that the machines are wrong, but that they are right too often, too cheaply, and too quickly for the institution behind them to absorb what they produce.

The proofs are correct, the claims valid, the code passes review. Underneath all of it sits the largest hidden assumption of all: that integration will somehow take care of itself. We are still hiring as though production were the constraint, and organising firms around the old world where production determined how much integration capacity we needed.

That is backwards in a world of AI abundance.

Look at your own organisation. If the roles, staff ratios and career paths have not changed, you are staffing for a constraint that has already moved.

About

I analyse AI progress beyond the headlines, focusing on enterprise execution, incentives, and real-world economic impact.