Why "AI 2040" Gets the AI Race Wrong!

Picture two runners at mile 21 of the same marathon.

Neither one can slow down first because slowing down means losing.

Now picture someone at the side of the road with a megaphone, telling both of them to stop and walk the rest of the way.

“Trust me. It’s for your own good.”

That’s the pitch behind AI 2040: Plan A, the document that’s been sitting in every group chat I’m in this past one week.

I disagree with the report massively. The biggest of it being that the AI race isn’t just a technology race.

It’s a game theory problem.

Now this isn’t the first time we’ve been here.

During the Cold War, there was the famous Doomsday Clock, tracking humanity’s proximity to catastrophe.

Every generation believes it’s living through the most dangerous moment in history.

More often, they overestimate their ability to predict how new technologies reshape society.

The Doomsday Clock kept moving. Humanity kept moving too.

That doesn’t mean the fears were irrational.

It means forecasting civilisation decades into the future is an extraordinarily difficult business.

Which brings us to AI 2040.

AI 2040 is a policy document.

It lays out multiple future scenarios, evaluates them against one another, and ultimately argues for a preferred path: Plan A.

Its possible futures look roughly like this:

Plan A: Slow down all AI research

Plan B: Fight China (a dynamic, boots-on-the-ground war)

Plan C: Burn the lead (hell naah!)

Plan D: Race to ASI (if we aren’t already there)

Plan S: Ban all AI research (Dune, huh?)

The scenarios are thoughtfully constructed.

The conclusions are where things begin to break down.

At the core of AI 2040 is a simple belief: If left unchecked, frontier AI could eventually become powerful enough to kill us all.

The implication is that governments should intervene before recursive self-improvement begins, especially if public opinion turns against AI.

This sounds reasonable.

Until you ask a much harder question.

How accurately has humanity ever predicted the second-order effects of transformative technologies?

The internet wasn’t expected to create social media. The smartphone wasn’t expected to become the primary interface to civilisation.

Every transformative technology arrives wrapped in uncertainty.

Not because experts are unintelligent.

But because complex systems don’t evolve linearly.

Predicting AI 15 years out with confidence is less like forecasting tomorrow’s weather and more like trying to predict TikTok after seeing internet work.

That doesn’t mean caution is wrong. Just the certainty with which it is being shared.

The document repeatedly treats AI like nuclear technology.

Hard to replicate easily and dangerous enough to kill our civilisation.

But history has shown that informational technologies spread far faster than material ones.

Britain once tried to cease the Industrial Revolution from spreading by restricting the export of textile machinery and skilled workers.

Within decades, the knowledge had crossed borders anyway.

The same pressures already exist in AI.

The technology to create LLMs is known.

We already have alternatives to the best of closed-weight models.

The technology, research, hardware and all the auxiliary markets will keep getting better with time.

Hence, trying to permanently contain frontier AI is less like containing uranium exports and technical know-how and more like trying to contain Linux — which was already open-source from the get-go.

The EU’s biggest challenge today isn’t talent. It continues producing some of the world’s best researchers and engineers.

Its true challenge is that its policy instinct often places a higher premium on minimising downside than maximising upside.

Every society chooses its own balance between innovation and precaution.

But history suggests that societies which consistently overweight caution rarely lead technological revolutions.

The steam engine was dangerous.

Electricity killed people.

Cars got into accidents (and even blew).

The internet enabled cybercrime.

Every general-purpose technology arrives with enormous externalities.

The answer has almost never been to stop building.

With LLMs Google even tried building them, but we know what happened.

The answer has always been to build better institutions around the technology.

That’s where AI 2040 feels backwards.

It spends more energy asking how to slow progress than how to govern it once progress becomes inevitable.

AI 2040 treats the concentration of frontier intelligence as one of the greatest risks facing humanity.

It’s a legitimate concern.

From the Mongols to the Romans to the Spaniards, technological superiority has always translated into geopolitical power.

And LLMs are the strategic technology of the 21st century.

But unlike guns and the previous technologies, intelligence isn’t a finite resource.

It scales.

The cost of copying intelligence trends toward zero.

Yes, frontier models are controlled by a handful of companies or countries.

But intelligence still diffuses.

Even the most centralised intelligence eventually reaches billions of people.

Usually behind a subscription.

The difference between the best closed-weight model and the best open-weight model isn’t the difference between a gun and a sword.

It’s closer to the difference between an AR-15 and an AK.

Which is precisely why the AI race isn’t equivalent to the nuclear arms race..

And setting it as such is a fool’s paradise.

The document gets one big thing absolutely right.

AI has stopped being just another software industry and has become geopolitical infrastructure.

Frontier models influence military capability, scientific research, economic competitiveness and national sovereignty.

Markets alone aren’t enough guardrails anymore.

That’s why proposals around international capability evaluations deserve serious attention.

Even Demis Hassabis has argued that independent evaluation of frontier systems could become an important part of global AI governance.

Because nobody knows if AGI will become dangerous.

Auditing frontier capabilities is fundamentally different from slowing all research.

One improves visibility while the other assumes visibility is impossible.

All policy is ultimately a compromise between ideals and reality.

Reality usually wins.

Every nation would probably prefer a world where everyone slowed down together.

The problem is that no nation can verify everyone else’s restraint.

States don’t optimise for absolute safety; they optimise for relative advantage.

The United States and the Soviet Union both understood that nuclear weapons could destroy civilisation.

Neither stopped building them because unilateral restraint was strategically irrational.

AI inherits the same incentive structure.

The genie isn’t just out of the bottle.

It’s already being copied and adopted.

The uncomfortable truth is that AI isn’t a problem we can vote away.

It’s a technology we’ll have to govern while it’s accelerating.

If you don’t build it, someone else will.

The real question is no longer whether AI will be built.

It’s whether nations choose to remain consumers of someone else’s intelligence — or build an intelligence stack of their own.

Agents work better as a team.

Hyperagent is where you set up specialist agents with defined jobs, shared context, and clear handoffs.

They run on schedules in the cloud, pass work to one another, and bring you in when a decision requires your judgment.

Hyperagent is giving away $200 in credits to This Week in AI’s readers!

Build your first agent today!