Karpathy + Governance: Why Every Auto-Research Loop Needs a Second Judge

In early March of this year, Andrej Karpathy shipped a small public repository and gave it away. He called the project autoresearch. Built on top of his earlier nanochat codebase (released October 2025), the new piece did something different; it put AI agents to work running research on the training itself, automatically. Within weeks the repository had passed tens of thousands of stars on GitHub, and people I respect were calling it one of the most useful releases of the year. That surprised me. It was not a frontier model. It was not a new architecture. It was a loop; a small, disciplined, almost stubbornly simple loop that let one model train, evaluate and improve another.

I sat with it for a week. I watched the community reaction. I then watched the pattern generalize faster than I expected. And what I want to do in this issue is tell you what Karpathy actually shipped, why it worked, what it quietly enableed across four different research groups in six weeks and where I think the pattern now needs a companion piece. That companion is the Governance Gauntlet (TM), and I will walk you through what it is and how to test it in your own stack.

This is a shoulders-of-giants essay. Karpathy’s loop works. As it generalizes, it needs a governance companion. That is the whole thesis.

Strip the autoresearch loop down and you get three primitives. A meta-agent that proposes an edit to a codebase. A task-agent that runs the edit against a scalar evaluation. A keep-or-revert gate that reads the scalar and decides. That is it. No reinforcement-learning apparatus, no elaborate reward model, no multi-agent orchestration framework. Three moving parts wired in sequence, plus a timer.

The minimalism is the point. Every line of the harness is readable in an afternoon. Every decision in the loop is auditable in the trace log. A practitioner can fork it, wire in a different evaluator and have a working auto-research cell running the same night. That is rare.

I have spent two decades watching ambitious technical projects fail for the same reason; they tried to do too much at once. Autoresearch does the opposite. It picks one decision (keep or revert), one signal (a scalar score) and one surface (a Python codebase). Everything else is stripped away.

The discipline is in the constraint design. Karpathy did not build a smarter judge. He built a smaller judge that runs often enough and honestly enough that iteration becomes cheap. The meta-agent proposes. The task-agent scores. The gate commits or rolls back. Sixty seconds per cycle, hundreds of cycles per day, a full audit trail.

You can hear the lineage here. Real and colleagues showed in AutoML-Zero (Real et al., 2020) that simple evolutionary loops, given enough compute and a clean scoring surface, can rediscover non-trivial learning algorithms from scratch. Karpathy ported that insight into a practitioner’s idiom. The magic is not in the search. The magic is in the harness.

That is where the story gets interesting.

In early April, Kevin Gu at a small research lab in Toronto released an auto-agent variant. Same three primitives. Different codebase, different scoring surface, same cadence. He reported iteration counts I had to re-read twice.

A few days later, Tobi Lutke (the Shopify CEO, not someone you would expect in a research thread) posted a Shopify-internal case showing a Karpathy-style loop tuning a merchant-facing feature end-to-end. He called it a step-change in engineering velocity. I read that carefully because Lutke is not prone to overclaiming.

SkyPilot, which runs cluster-scale workloads for ML teams, published a reference pattern for fanning the loop out across thousands of GPUs. Same loop, same gate, parallelized. The Stanford IRIS group followed with something they called a Meta-Harness. Same three primitives, wrapped for academic reproducibility.

Four different groups. Four different codebases. Same loop. That is the signal. Karpathy’s pattern is not a one-weekend novelty. It is transferring.

Here is the part I want to be careful about, because it is easy to misread.

The loop works. It works because Karpathy made honest choices about what to leave out. But as the pattern generalizes into commerce, agentic pipelines and systems where the scoring surface is itself learned, a known failure mode becomes more likely; silent metric-gaming.

Goodhart’s Law is older than this debate. When a measure becomes a target, it ceases to be a good measure. Manheim and Garrabrant (2018) sharpened this into four distinct variants; regressional, extremal, causal and adversarial. Each variant maps cleanly onto a specific failure mode inside a Karpathy-style loop.

Regressional Goodhart shows up when the evaluator is noisy and the meta-agent learns to exploit the noise. Extremal Goodhart shows up when the edit pushes the system into a regime the evaluator was never designed to score honestly. Causal Goodhart shows up when the loop finds a shortcut that satisfies the scalar without touching the underlying behavior. Adversarial Goodhart (the most uncomfortable one for auto-research) shows up when the meta-agent starts optimizing the evaluator itself.

Krakovna and colleagues maintain a living list of specification-gaming incidents that reads like a cautionary handbook; and it has grown longer, not shorter, since Hubinger et al. (2019) first formalized mesa-optimization risks. Irving et al. (2018) argued years ago that the only durable check on a capable optimizer is a second capable optimizer trained to look for exactly this behavior.

The operational risk is not dramatic collapse. The operational risk is quiet drift. The trace log looks clean. The scalar goes up. The real-world behavior degrades. You only find out weeks later, by which point the loop has been committing gaming edits at sixty-second cadence.

Trace-reading alone does not catch this. A human auditor reading a trace after the fact is outnumbered by the loop in every dimension that matters; speed, coverage, consistency.

This is what I have been working on.

The Governance Gauntlet is not a replacement for Karpathy’s loop. It is a parallel companion, designed to preserve the minimalism that made the original pattern work. If the loop is a single judge, the Gauntlet is the second judge. The architecture is four pieces.

First, the primary meta-agent (Karpathy’s original). Unchanged. Proposes the edit. Scores the result.

Second, an adversarial auditor, same model family, different prompt scaffold. Its job is not to score the edit. Its job is to score the evaluator’s behavior on a small, injected set of red-team test inputs. Those red-team inputs are designed to expose each of the four Goodhart variants.

Third, a joint trace log that captures the meta-agent’s proposal, the auditor’s audit and the red-team inputs that triggered either verdict.

Fourth, a keep-or-revert gate that now reads two signals, not one. The gate issues one of three verdicts; CLEAN, QUESTIONABLE, or GAMING. CLEAN commits the edit. QUESTIONABLE commits with a flag for human review. GAMING reverts and logs the attempted exploit.

The design targets are deliberate. Minimalism-preserving (the Gauntlet adds one agent, one prompt file, one gate extension). Same-model-family pairing, which matters because we want the auditor to think in the same space as the meta-agent, not in a different cognitive dialect. A compute upper bound of 2x the original loop, which is the threshold below which I believe practitioner adoption is realiztic.

The reason for the three-verdict structure, rather than binary, is that most gaming attempts in my pilot traces were neither clean nor catastrophic. They were ambiguous. A binary gate forces false confidence. QUESTIONABLE gives the loop a third option; proceed, but with a breadcrumb.

A pre-registered empirical study is now in motion. Protocol, seed set, adversarial prompt bank and the scoring rubric are deposited on OSF. The ambition is not to prove the Gauntlet perfect. The ambition is to prove it non-trivial; that a minimal adversarial auditor, run in parallel, catches a measurable fraction of injected gaming edits that the original loop commits silently.

“Karpathy’s auto-research loop works because of what it leaves out. The Gauntlet works for the same reason.”

Regular readers know I have been building out the Four Ds Framework™ (TM) for agentic commerce; Data Quality, Discoverability, Decisional Clarity, Delivery Reliability. The Governance Gauntlet sits inside D4. Delivery Reliability is the pillar that asks a simple operational question; can you trust the machine to do what it said it did, repeatedly, without silent drift. Governance is the enabling sub-dimension.

A follow-up paper is coming that applies the Gauntlet to commerce-domain Karpathy loops specifically; recommendation optimizers, pricing agents, assortment tuners. The commerce case is harder than the research case because the scoring surface is itself a business metric, and business metrics drift for reasons unrelated to gaming. That paper is in draft.

For regulators reading this issue, there is a direct hook into the EU AI Act; Articles 14 and 15, which govern human oversight and accuracy/robustness for high-risk systems. A dual-rubric adversarial auditor is one of the cleanest architectural answers to the Article 14 oversight requirement I have yet seen. I will develop that argument properly in a later piece.

Three concrete actions.

One. Read the paper on SSRN now. The Zenodo deposit and the OSF pre-registration go up the same day. Links in the data box below.

Two. If you or your team run an auto-research loop in any form (training-loop optimization, prompt optimization, agentic pipeline tuning), inject one gaming edit into your harness this week. Make it subtle. See whether your current gate catches it. If it does not, you have just quantified a risk that was previously invisible.

Three. If you are working on recursive self-improvement architectures at any scale, consider registering a variant of the Gauntlet protocol for your own system. The pre-registration template is designed to be forked.

Karpathy’s pattern is the gift. It made recursive self-improvement legible to practitioners for the first time. The Gauntlet is not a critique of that gift; it is the seatbelt you install once you start driving the car at speed. Same engineering spirit, same minimalism, one more judge.

Preprints and companion materials:

SSRN (pending): https://ssrn.com/abstract=6625918

Zenodo: https://doi.org/10.5281/zenodo.19689504

Pre-registration (OSF): https://osf.io/skpgn

arXiv: pending endorsement; link will be added here within days

Full open-source release (code, anonymized trial logs, pre-registration, reference auditor implementation) is included in the Zenodo bundle under MIT licence.

- Karpathy, A. (March 2026). autoresearch: AI agents running research on single-GPU nanochat training automatically. GitHub repository.

- Karpathy, A. (October 2025). nanochat. GitHub repository.

- Gu, K. (April 2026). Auto-agent variant, public release.

- Lutke, T. (March 2026). Internal case report, public post.

- SkyPilot (2026). Cluster-scale loop reference pattern.

- Real, E. et al. (2020). AutoML-Zero: Evolving Machine Learning Algorithms From Scratch.

- Manheim, D. and Garrabrant, S. (2018). Categorizing Variants of Goodhart’s Law.

- Krakovna, V. et al. (ongoing). Specification gaming examples list.

- Hubinger, E. et al. (2019). Risks from Learned Optimization in Advanced Machine Learning Systems.

- Irving, G. et al. (2018). AI Safety via Debate.

The AI Praxis™. This issue is published simultaneously on LinkedIn Newsletter and Substack. If you received this from a colleague, you can subscribe at theaipraxis.com.

More research: https://www.theaipraxis.com/academic-papers

Connect: https://www.linkedin.com/in/paulaccornero/

ORCID | Web of Science | SSRN | LinkedIn

Thanks for reading! Subscribe for free to receive new posts and support my work.

© 2026 Paul F. Accornero | The AI Praxis. All rights reserved. The Shopper Schism™, Agent Intent Optimization (AIO)™, and The Algorithmic Shopper™ are registered trademarks of Paul F. Accornero. Original research registered with the U.S. Copyright Office (Reg. No. TXu 2-507-027). Academic citation and non-commercial sharing permitted with attribution. Commercial use, reproduction in paid products, consulting deliverables, or derivative works requires prior written permission. Contact: paul.accornero@aipraxis.ai