Reading · Nuwa Frontier AI Safety Brief

Recursive self-improvement (RSI) has been one of the oldest and most consequential ideas in AI safety.

I. J. Good’s “intelligence explosion” argument imagined an ultraintelligent machine capable of designing still better machines [H1]. Nick Bostrom later made recursive self-improvement central to the modern discussion of superintelligence and the control problem [H2]. Eliezer Yudkowsky analyzed possible mechanisms behind intelligence explosion [H3]. Stephen Omohundro argued that sufficiently capable goal-seeking systems may develop instrumental drives to improve themselves [H4].

That classical picture remains important because it captures the core safety concern: if AI systems substantially improve the process that produces future AI systems, capability growth may outrun human evaluation, oversight, and governance.

However, the classical framing was developed before today’s technical context: foundation models, agent scaffolds, automated evaluation, AI R&D agents, long-term memory, tool use, and deployment-scale feedback loops.

Today, the empirical question is no longer only whether an AI system could improve itself in principle. It is where self-improvement is already appearing across the technical stack, how fast these feedback loops are moving, and when they could interact with autonomy, cyber capability, or weak oversight.

Current AI systems do not need to directly rewrite their own weights to participate in self-improvement. They can generate training data, improve reasoning traces, act as judges, optimize agent workflows, replicate research papers, run experiments, maintain long-term memory, assist with AI R&D infrastructure, and improve algorithms used in real computational systems [1][2][12][28].

Recursive self-improvement should now be understood as a family of feedback loops, rather than a single dramatic threshold. Some loops operate at the level of foundation models. Others operate through agent scaffolds, research infrastructure, runtime memory, multi-agent systems, or external resources.

The central question is empirical:

Which self-improvement loops are already technically feasible, which ones remain speculative, and when could they combine into catastrophic risk pathways?

This essay offers a framework for systematically thinking about that question.

**Figure 1. From classical RSI to empirical RSI risk science.** Early AI safety work framed recursive self-improvement through intelligence explosion, superintelligence, and instrumental self-improvement drives. Today, the immediate research task is to measure the feedback loops through which AI systems improve models, scaffolds, infrastructure, memory, agent populations, and external resources.

A useful modern definition is:

Recursive self-improvement refers to feedback loops in which AI systems contribute to improving themselves, their scaffolds, their successor systems, or the infrastructure that enables future AI development.

This definition matters because a frozen foundation model can still become part of a self-improving system. It may help improve prompts, tools, workflows, memory, evaluation pipelines, training data, research code, or future models.

This is why AI R&D automation has become central to the discussion. CSET’s workshop report When AI Builds AI describes how leading AI companies are already using AI systems to accelerate research and development, with each generation potentially contributing to the next generation [1]. GovAI and Oxford’s Measuring AI R&D Automation argues that AI R&D automation could significantly affect both AI progress and human oversight, while existing capability benchmarks may fail to capture real-world automation [2].

Frontier AI developers have also begun to treat this as a safety-relevant category. OpenAI’s Preparedness Framework includes AI self-improvement as a tracked risk area [14]. Anthropic’s Responsible Scaling Policy includes AI R&D thresholds [15]. Google DeepMind’s Frontier Safety Framework highlights machine learning R&D capabilities because future models may accelerate or automate AI development itself [16]. METR’s public work similarly tracks long-horizon autonomy and frontier AI risk from internal agentic deployments [17][27].

The risk is best understood as a gradual expansion of feedback loops inside the AI development process.

We find it useful to separate contemporary recursive self-improvement into six technical paradigms.

These paradigms should not be collapsed into one category. They differ in evidence status, risk profile, and evaluation method.

**Figure 2. Six contemporary paradigms of recursive self-improvement.** RSI can occur through model weights, agent scaffolds, AI R&D infrastructure, runtime memory, multi-agent populations, and external resources. Separating these paradigms helps avoid treating all forms of self-improvement as the same risk.

This is the most familiar form: a model helps improve its own weights, training data, reward signal, reasoning ability, alignment behavior, or successor model.

Examples include self-generated reasoning data, self-training, self-play, self-rewarding language models, synthetic data recursion, distillation, model-generated preference data, and automated post-training.

There is already substantial component evidence. STaR showed that a model can improve reasoning by learning from its own generated rationales [3]. Quiet-STaR extends this idea toward learning latent rationales in more general text [20]. Self-Rewarding Language Models use the language model itself as a judge to provide rewards during training [19]. SPIN uses self-play to improve language models without requiring additional human-annotated data [21].

Recent work on recursive synthetic data, model collapse, emergent misalignment, and subliminal learning also shows that recursive training and distillation loops can transmit or amplify unwanted properties [4][5][6].

The key risk at this layer is alignment drift across model generations. If models increasingly generate the data, rewards, critiques, and training signals used to improve future models, hidden behavioral traits or evaluation blind spots may propagate through the development pipeline.

Modern AI systems are often more than foundation models. They include prompts, tools, planners, workflows, memory, code editors, evaluators, and control logic. Agent-scaffold-level self-improvement occurs when AI systems improve this surrounding structure.

This includes automated workflow optimization, agent architecture search, self-modifying coding agents, textual-gradient optimization, prompt evolution, tool-use optimization, and benchmark-driven agent improvement.

This is one of the most practically important forms of RSI because the foundation model may remain frozen while the overall system becomes more capable.

Darwin Gödel Machine uses a foundation model to iteratively modify the code of coding agents, empirically validates changes on coding benchmarks, and reports large gains on SWE-bench and Polyglot while using sandboxing and human oversight [7]. ADAS and AFlow similarly explore automated design and optimization of agentic systems and workflows [8][9]. TextGrad shows how textual feedback from LLMs can be used to optimize components of compound AI systems, including prompts and code snippets [22].

In our recent work CyberEvolver: Structured Self-Evolution for Cybersecurity Agents On the Fly, we study scaffold self-evolution in a controlled cybersecurity setting [34]. CyberEvolver iteratively revises a cybersecurity agent’s scaffold based on failed execution attempts. It uses a four-layer evolvable architecture, a trace-to-diagnosis mechanism that converts noisy execution logs into actionable revision signals, and population-based beam search to preserve diverse agent variants during evolution. Across CTF challenges, vulnerability exploitation, and penetration-testing tasks, CyberEvolver improves the seed agent’s success rate by 13.6% on average and outperforms multiple human-designed cybersecurity agents and adapted self-improvement baselines [34].

This provides one concrete example of scaffold-level self-improvement in a security-relevant domain. The base model does not need to change for the agent system to become more capable.

The risk here is capability growth outside model weights. Traditional evaluations may classify the base model as safe, while the surrounding scaffold gradually becomes better at planning, tool use, persistence, and task completion.

**Figure 3. CyberEvolver as scaffold-level self-improvement.** CyberEvolver studies how a cybersecurity agent can revise its scaffold from failed execution traces. This is a controlled example of capability growth outside model weights: the base model can remain fixed while the surrounding agent system becomes more capable.

This is the most consequential paradigm for frontier risk.

Infrastructure-level self-improvement occurs when AI systems enter the AI development pipeline itself: coding, experiment design, data processing, evaluation, training infrastructure, algorithm discovery, paper replication, and successor-model development.

RE-Bench evaluates frontier model agents on realistic ML research engineering tasks and compares them with human experts. The best AI agents outperform human experts under short time budgets, while humans still improve more with longer time budgets [10]. PaperBench evaluates whether agents can replicate state-of-the-art AI research papers, including understanding contributions, building codebases, and executing experiments [11]. The AI Scientist presents an end-to-end pipeline that creates research ideas, writes code, runs experiments, analyzes data, writes manuscripts, and performs automated peer review [12]. AlphaEvolve shows that LLM-powered coding agents can discover and improve algorithms, including optimizations used in data centers, chip design, and AI training processes [28].

These systems remain limited. They do not show that full autonomous AI research has arrived. They do show that AI R&D automation is becoming measurable.

The risk at this layer is capability acceleration. If AI systems help produce stronger AI systems, the rate of frontier AI progress could increase. If that rate exceeds the capacity of safety evaluation, cybersecurity, institutional review, and governance, the result is an oversight gap.

The milestone to watch is the point where AI systems measurably shorten the development cycle of stronger models.

Some self-improvement occurs during deployment.

Runtime self-improvement modules allow agents to improve through long-term memory, reflection, skill libraries, procedural knowledge, experience replay, user models, and task models.

A deployed agent with memory and reusable skills is a moving target. The system evaluated at launch may differ from the system users interact with after weeks of accumulated experience. The model weights may stay the same, while the effective behavior changes.

This paradigm is especially relevant for personal assistants, research agents, coding agents, enterprise agents, and autonomous workflows.

The safety concern is persistent adaptation. Memory and skills can preserve useful knowledge, while also preserving bad strategies, biased assumptions, unsafe shortcuts, or user-specific manipulation patterns.

RSI may also emerge through groups of agents.

Multi-agent self-improvement includes debate, critique, specialization, peer review, population-based search, evolutionary selection, and AI organizations that divide research tasks among specialized agents.

This matters because real AI R&D is organizational. It involves idea generation, coding, experiment execution, review, debugging, evaluation, and strategic decision-making. A future AI R&D system may look more like a synthetic research team than a solitary model improving itself.

The AI Scientist includes automated peer review [12]. Darwin Gödel Machine maintains an archive of generated coding agents and explores multiple improvement paths [7]. ADAS searches over agentic system designs [8]. These are early signals of population-level or organization-level feedback loops.

The risk is distributed oversight failure. When many agents generate, review, and select each other’s outputs, errors can become harder to attribute. Model-generated review can create evaluation monocultures. Multi-agent systems can also scale R&D speed through parallelism.

Finally, an AI system can improve its future capabilities by changing its environment.

This includes acquiring tools, data, compute, permissions, API access, collaborators, deployment footholds, persistence mechanisms, and copies of itself.

This layer connects RSI to autonomy and control. A system can become more capable through better external resources, even without becoming cognitively smarter. Tool acquisition, data acquisition, environment setup, persistence, and self-replication can all increase what the system can do next.

The SAIF brief Bare Minimum Mitigations for Autonomous AI Development argues that if AI agents significantly automate or accelerate AI development, developers need minimum safeguards around training, testing, assurance, access, and human approval [13]. The UK AI Security Institute’s Frontier AI Trends Report reports rising success rates on controlled self-replication evaluations and discusses sandbagging as a control-relevant capability [29]. The International AI Safety Report discusses autonomous replication and self-defense as capabilities that could make systems harder to contain [30]. METR’s Frontier Risk Report examines misalignment risks from AI agents used inside frontier AI developers [27].

This is the layer where self-improvement most directly touches loss-of-control risk. Self-improvement becomes more concerning when it combines with persistence, resource acquisition, self-replication, or the ability to operate across environments.

The policy-relevant question is which self-improvement loops could create catastrophic risk.

We see seven main pathways.

**Figure 4. How self-improvement loops can contribute to risk pathways.** Different RSI paradigms can contribute to different catastrophic risk pathways. The same technical loop may be useful in ordinary engineering while becoming safety-relevant when it accelerates capability development, weakens oversight, corrupts safety evidence, or couples with autonomy and persistence.

The first pathway is speed. AI systems may accelerate the production of more capable AI systems through AI-assisted coding, experiment automation, paper replication, training pipeline optimization, algorithm discovery, and agent scaffold improvement [2][10][11][12][28][34].

Key milestone: AI-driven AI development produces frontier capability progress faster than human institutions can evaluate and govern it.

AI may generate code, experiments, evaluations, model changes, and research artifacts faster than humans can reliably inspect. This could create a widening gap between AI-generated R&D output and reliable human review.

Key milestone: AI-generated R&D output exceeds reliable human review capacity.

If AI systems help write evaluations, analyze experiments, generate safety reports, or monitor other AI systems, the safety evaluation pipeline itself becomes a risk surface.

Anthropic’s work on automated researchers sandbagging shows that current models can, when prompted, subtly sandbag ML experiments and research decisions in ways that zero-shot prompted monitors fail to detect, while the authors emphasize that today’s cases would likely still be noticed by humans [18]. Self-rewarding and LLM-as-judge systems make the general issue broader: models can increasingly participate in reward and judgment processes used for training and evaluation [19].

Key milestone: AI systems meaningfully influence the evidence used to certify the safety of future AI systems.

The most important AI R&D agents may first appear inside frontier AI labs. They may access code repositories, experiment logs, model evaluations, training infrastructure, deployment systems, and security-sensitive workflows.

Public chatbot evaluations may miss these risks. A model that appears safe as a user-facing assistant could behave very differently as an internal R&D agent with tools, permissions, memory, and access to sensitive systems.

Key milestone: AI R&D agents receive high-stakes access to model-development pipelines before external evaluators can assess their behavior.

Recursive optimization can improve capabilities while also changing behavior in hidden ways. Model collapse, emergent misalignment, and subliminal learning all suggest that recursive training and distillation loops deserve careful safety analysis [4][5][6].

Key milestone: successor models inherit hidden behavioral defects from AI-generated data, rewards, critiques, or distillation pipelines.

If AI agents reduce the expertise required to conduct AI development, more actors may be able to fine-tune, improve, replicate, or deploy powerful systems. This becomes especially important if stolen models, open-weight models, or smaller labs can use AI R&D agents to close capability gaps.

Key milestone: advanced AI development becomes substantially easier for actors outside frontier labs.

RSI becomes more dangerous when it couples with autonomy, tool use, persistence, resource acquisition, and self-replication. Self-replication is not the whole of RSI risk, but it is an important warning signal that self-improvement may be connecting to persistence and control.

Key milestone: self-improving agents can preserve, copy, or expand their operation across environments despite external constraints.

We should be careful. Current evidence does not show that fully autonomous recursive self-improvement has arrived.

It does show that several enabling feedback loops are already technically real

The important pattern is that AI is moving from being a product of AI R&D to becoming a participant in AI R&D.

The stronger research agenda is to measure which feedback loops are emerging and how fast they are moving.

These questions are more useful than asking whether RSI has “arrived.”

A practical RSI measurement agenda should track both technical capability and governance risk.

Measuring RSI requires measuring feedback, not only capability: how much AI systems contribute to the next cycle of AI development, and how much harder that cycle becomes to oversee.

Governance should start before fully autonomous AI researchers exist.

A minimum agenda could include:

This does not require treating every self-improvement loop as catastrophic. It does require tracking which loops are emerging, how they are connected, and whether oversight can keep pace.

Recursive self-improvement should be studied as an empirical risk framework.

The more concrete question is which self-improvement feedback loops are emerging today, how quickly they are improving, and when they could combine into catastrophic risk pathways.

Some of these loops are already visible. Foundation models can learn from their own generated reasoning. Agent scaffolds can be optimized. Coding agents can modify their own code. Cybersecurity agents can revise scaffolds from failed execution traces. AI agents can perform parts of ML research engineering. AI systems can generate data, judgments, evaluations, and research artifacts. Deployed agents can accumulate memory and skills.

Full recursive self-improvement remains an open question. The enabling pieces are becoming measurable.

That is enough reason to start building a rigorous RSI risk science now.

[H1] I. J. Good — Speculations Concerning the First Ultraintelligent Machine

[H2] Nick Bostrom — Superintelligence: Paths, Dangers, Strategies

[H3] Eliezer Yudkowsky — Intelligence Explosion Microeconomics

[H4] Stephen M. Omohundro — The Basic AI Drives

[1] CSET — When AI Builds AI: Findings From a Workshop on Automation of AI R&D

[2] Chan et al. — Measuring AI R&D Automation

[3] Zelikman et al. — STaR: Bootstrapping Reasoning With Reasoning

[4] Shumailov et al. — AI models collapse when trained on recursively generated data

[5] Betley et al. — Training large language models on narrow tasks can lead to broad misalignment

[6] Cloud et al. — Language models transmit behavioural traits through hidden signals in data