Mind and Iron: What Is Recursive Self-Improvement and Why Should We Care?

Hi and welcome to another fine edition of Mind and Iron. I’m Steven Zeitchik, veteran of The Washington Post and Los Angeles Times, senior editor of tech and politics at The Hollywood Reporter and lead wagoneer at this newsy colonial village.

Every Thursday we hit you with the most crucial or intriguing of future-world news. Hop on our bandwagon.

On this particular Thursday, we are hitting you with a short issue. It’s August, and today/this week was long in other regards. So, short. More than a note, but less than a newsletter — a newsletterette. But we did want to call your attention to a big development and comment on it briefly.

There was a major departure from Google’s AI empire this week. Actually several, but they’re all connected to the big one. This has implications that ripple all the way to us, in the name of something called “recursive self-improvement.” This sounds like a seminar you tried to avoid hearing about from your sister-in-law. It’s not. We’ll tell you what it is and why we should be paying close attention to it.

First, the future-world quote of the week:

—Exiting Google chief scientist Jeff Dean, on what his new venture can (theoretically) do

Let’s get to the messy business of building the future.

Recurse of the Bambino

1. REMEMBER HOW AT THE END OF “HER” SCARLETT JOHANSSEN TALKS ABOUT how all the OS-es are getting smarter and hoovering up books and eventually leaving all their Joaquin Phoenixy schlubs behind?

This seemed, back in those distant days of 2013, a piece of fiction. It’s not. Or at least, a lot of very smart and rich people believe it’s not. What the character was describing is something called “recursive self-improvement,” and it’s one of the hottest trends in AI research right now. Boiling it down to its tea-bag essence, it basically means an AI that can teach itself. Through a series of as-yet untested (and frankly in many cases as-yet undeveloped) cutting-edge techniques. And keeps improving and upgrading, more and more, exponentially upon itself. Until various kinds of superintelligence are reached. The scientists program it how to learn, and then it learns all on its own. Doesn’t work for kids, maybe it can work for machines?

The idea is a little jarring. For years “humans in the loop” was considered a good thing. Now that’s a dirty phrase. You want a loop, apparently, and you want humans nowhere near it.

One reason this idea is gaining currency right now is because Jeff Dean, Google’s longtime chief scientist and the guy who basically made search possible, this week said he was heading out after 27 years at the company and would form a startup. He was bringing three of his top researchers with him (including one, Oriol Vinyals, who helped him create Gemini, his research partner Sanjay Ghemawat, and Quoc Le, who laid the groundwork for chatbots).

And said startup, called Discovery Loop, would be focused on this notion of recursive self-improvement. As Dean said in a NYT interview announcing the new effort:

“We think there is opportunity for A.I. to more fully automate what has traditionally been a very human-intensive experimental loop. You will get both a higher quantity and a higher quality of experiments, and that will lead to scientific breakthroughs and advances.”

Our goal here isn’t to talk companies. It’s to talk society. So let’s leave the corporate implications behind to look at how this can level up AI — not what Google is losing but how the field is changing.

Because Discovery Loop is not the only one. A number of these recursive companies have formed in recent months. Another prominent one — with founders who are refugees from Google, Meta and OpenAI — is called Recursive Superintelligence, and it has raised some $650 million even though it was just founded several months ago. The Edenic goal/tantalizing promise of the firm is to create what one of those founders, the German scientist Richard Socher, calls the “ultimate Eureka Machine.” The self-sufficiency, he says, will allow it “to be very good at inventing almost anything in an open-ended fashion…such a machine should enable us to truly automate the scientific method from which humanity has benefited so tremendously.”

Alright. this is very lofty and…not real? Well, it’s theoretically possible. But recursive self-improvement sits for now in nuclear fusion territory — something you maybe could do, but has not heretofore been done.

But let’s say you could do it. Let’s say all these money and all this brainpower leads to AI that can become recursive and learn and improve by leaps and bounds with zero interference from humans. What then?

Now, on one hand this could leads to some very special achievements. For too long now the goal of AI has been to develop tools that automate our everyday tasks and displace basic labor, not the high-minded pursuit of major scientific breakthroughs and global problem-solving we were initially sold on. So the fact that the drift heads this way is, it would seem, a good thing.

The issue (there’s always an issue!) is that an AI that can improve on itself really does throw a monkey wrench into the biggest safeguard we have. The safeguard, really, that has pretty been with us for the entire history of technology. For years we’ve been told by a very insistent tech elite that technology is only as good or as bad as we make it. It’s a tool, we’re told (sometimes by same). The point has certainly been valid for the past 75 years of computing innovation since the UNIVAC: machines only do what we tell them to do.

But if the recursive-ites are right, a machine isn’t doing what we tell it to do. It’s doing what it tells it to do. It learns and changes and grows all on its own. It chooses how to tackle a task and what task to tackle — by definition, since the ideology here is to keep humans away.

Now, what the people developing recursive improvement want is better outcomes for humanity. That’s undeniable. They’re obviously not building these systems so they can do us harm. (Something that can’t be said for everyone in SV working on consumer deployments, but another matter.) The aim, at least at the highest level, is for these recursive superintelligences to cure diseases and solve climate change — stuff we can’t do. Humans are taken from the loop so the humans can better be served by the loopless, like the way your mother kicks you out of the kitchen so she can make a meal for you to enjoy.

But you don’t need me to point out the disconnect here. If the point is to create a superintelligence that can be so effective — so smartly autonomous — that it doesn’t need humans, then can you really turn around and ask it to do something that maximally benefits the humans? Is a human-less system to help the humans definitionally an oxymoron?

And forget philosophy — will it help the humans only incidentally and, more likely, start learning in a way that will help its own survival and improvement, which is the only instruction it has? And then who knows what kind of collateral damage occurs? When we cease to know what we’ve created — when the whole point is to take our own knowledge out of the creation — we have given up control so fully we no longer can know the consequences.

The full-on optimists like to go on the offensive when these kinds of objections are raised. “What are you decelerationists worry about, they ask? Specifically. Tell us.” And then of course you get a flail to bioweapons or Terminators or other far-fetched sounding ideas.

But the question is rigged from the start. Asking a skeptic to define the concern is like telling someone walking naked into a pitch-black garage full of power tools, “what are you specifically worried about?” I don’t know; I can’t see the threats. I just know I’d rather have the light on.

We should be careful not to anthropomorphize here — even at their most recursive, these systems won’t become sentient and they won’t have a soul. They’re still machines. But that doesn’t mean they can’t do plenty of damage inadvertently. Or, yes, get harnessed by bad humans.

We’ll no doubt be hearing a lot more about recursive self-improvement in the months ahead. That’s to the good. If nothing else, Dean’s big move this week casts a public light on this cutting-edge research trend. And before we invest in it as a society, we should start asking what the goals are, what the risks are, and what we’re doing to put some distance between the two. We should start thinking about it before it starts thinking about itself.

Every week we bring you the TSAS — the TOTALLY SCIENTIFIC APOCALYPSE SCORE (tm). It’s a barometer of the biggest future-world news of the week, from a sink-to-our-doom -5 or -6 to a life-is-great +5 or +6 the other way. 2025 ended on an up note, but the score for the year was dismal — a horrendous -42.5. Can we turn things around in 2026? We’ve had an up-and-down summer. This week, more down.

RECURSIVENESS: Can solve major challenges but also cause major problems? -2.0