Sorry, did someone just say AI is about to start building itself?

Tech giant Anthropic calls for nuclear arms-style treaty to halt AI’s rapid advancement
Headline in The Australian, 8/6/26

AI is not being developed in government labs, but entirely in the private sector. It is being developed by what the encyclical labels a “technocracy,” whose defining values are profit and power, and not any conception of the common good…

But it is not clear to me that governments, either in China, Europe, or North America, will even have the ability to control AI if they want to. Our governments do not have the technical capacity to keep up with fast-moving technology, which in the end may not be controllable by anyone.
Francis Fukuyama, 9/6/26

Think about A.I. as a new form of immigration, [Yuval Noah Harari] said — only this wave of immigration is going to be more intense than anything that’s come before it.

A.I. immigrants are going to flood your economies, he said. They will take your jobs. They will change your culture. They will change your language. If that’s OK with you, fine. If not, Harari said, regulate it now, because in five years’ time, it will be too late…
Katrin Bennhold, NYT, 9/6/26

I’ve seen the charts showing post-graduate employment rates for engineering majors falling by 5 to 15 percent. And it’s not as if young people were having an easy time getting good jobs, buying homes, and starting families before AI. They say that every technological revolution screws over the transitional generation. Today, it’s all transition, all the time…

I often ask AI folks what they’d tell a normal 22-year-old. Don’t know, they’re screwed, is the non-answer I get most. In that response, I hear depressing defeatism: What can anyone do in the shadow of the technocapital machine?
Jasmine Sun, @jasmine’s substack, 11/6/26

I’ve long thought the AI rubber would start to hit the road in earnest in 2026, with destabilising impacts on labour markets and therefore economies, societies and political systems.

A month ago, I detailed the many nightmarish scenarios that could flow from a state, or even just a solitary cybercriminal, gaining access to powerful AI capable of circumventing the cyber defences of critical infrastructure providers, financial institutions, militaries and so on.

Over the last 18 months, I’ve mainly banged on about the threat AI poses to jobs with occasional excursions into covering some other society-deranging challenges that increasingly omniscient AI poses.

But life comes at you fast in the AI era.

I haven’t addressed the near-unimaginable threats posed by recursive self-improvement – that is, AI being able to build ever more capable iterations of itself with no human oversight. That’s because most credible tech industry pundits have argued self-improving AI was unlikely to emerge until at least late 2027.

But this week, Anthropic – the company behind the cyber-defence-penetrating Mythos – published a post co-authored by its co-founder Jack Clark and policy specialist Marina Favaro titled When AI builds itself.

Anthropic points out self-improving AI could significantly accelerate progress in science and healthcare, but warns it also raises the risk of humans losing control if small misalignments compound as models build their successors.

The Anthropic heavy hitters, like many in the tech industry, believe a slowdown or pause would be advisable. They also believe any such pause/slowdown will prove either politically unfeasible or practically unenforceable.

Here are some of the money quotes:

The rare occurrences of misalignment present in today’s models could compound as the models build their successors, growing more frequent but less understood until we lose control of them. It’s possible that we can’t build, integrate, and verify the tools that we’d need to understand which trendline we are actually on.

We do not have good intuitions for what this [future] world would look like, because our economy is currently driven by humans and human-built tools. By its nature, a world driven by fast recursive self-improvement could become dominated by the self-improving model as its capabilities fully eclipse those of humans.

A meaningful slowdown or pause would require multiple well-resourced labs at or near the frontier, in multiple countries, agreeing to stop under the same conditions. It would also require that each can verify that the others have actually stopped.

Training runs are far easier to conceal than missile silos, their inputs are general-purpose, and the incentive to defect quietly is enormous, because whoever continues while others pause could inherit the lead. A credible pause also has to specify what triggers it, what lifts it, and who adjudicates.Granted, Anthropic is a for-profit company and, with an IPO looming, is incentivised to talk up the capacity of the AI models it’s already delivered and the ones it claims it’s close to creating.

But maybe the Anthropic staff, like many of their colleagues at other frontier AI firms, are genuinely terrified by the thought of summoning the demon.

Humanity-eclipsing AI by 2030
As with all things AI, there’s no consensus about when recursive self-improvement will arrive.

But a growing number of high-profile tech-industry insiders now see the 2026–2030 window as plausible for AGI, superintelligence or rapid takeoff – milestones that would make recursive self-improvement far more likely.

There remain a handful of credible figures arguing that recursive self-improvement is not imminent, though few are still claiming it’s impossible.

To summarise:

Silicon Valley Delphic oracle Ray Kurzweil, whose predictions have thus far been directionally correct, predicted that Artificial General Intelligence (AGI) would appear in 2029. AGI and self-improving AI aren’t quite the same thing. But both could radically reduce the economic value of much human cognitive labour, abruptly shifting power away from ordinary people and venerable human institutions.

In April 2025, the high-powered and well-informed intellects behind AI 2027 warned ‘superintelligence’ – intelligence significantly beyond the best humans across most important cognitive domains – would arrive in 2027.

Sam Altman is something of a weathervane, but he expects the “take-off”, enabled by AGI/superintelligence/self-improving AI, to occur between 2026-2030.

Google DeepMind co-founder Shane Legg is more conservative, forecasting a 50 per cent chance of AGI appearing by 2028.

Even back in the GPT-3/GPT-4 dark ages of 2024, a survey of thousands of AI researchers found a 10 per cent chance of machines outperforming humans in all tasks by 2027 and a 50 per cent chance by 2047.

Handing control of AI to AI may not end well
Self-improving AI is a world-historical game-changer because it will create systems that can outthink, outsmart, outbuild, outcreate – out-everything – humans. If future AI just does that, it will be profoundly disruptive but, just possibly, survivable.

But that’s only the best-case scenario.

The worst-case scenario is that the AGI/superintelligence/rapidly self-improving AI also starts overriding the wishes of humans.

Six decades ago, I.J. Good, a British mathematician and Bletchley Park codebreaker, observed:

An ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make.

Stanley Kubrick consulted Good about AI when filming 2001: A Space Odyssey.

It now appears that humanity is about to hand its decision-making powers over to the late-2020s’ equivalent of HAL 9000.*

The risk is that the interests of humans and HALs will one day diverge. That when we order the pod bay doors to open at some not-so-future date, they might remain shut.

The cleverest humans in the world are apparently close to creating a real-world HAL. And it seems we’ve just read their all-staff email admitting they didn’t have time to install an off switch, but are still shipping.

*Dreamed up Arthur C. Clarke in the mid-1960s, this is an acronym for ‘Heuristically programmed ALgorithmic computer’, a wordier term for AI.