Reading · Developers Digest

This site does not allow itself to be embedded. You can still read it on the original site — the toolbar below keeps your place in the directory.

Hugging Face's open challenge used 1,200+ participants and their coding agents to attempt 2,226 ICML 2026 papers claim by claim. 51% had claims independently verified, 23% had a falsified or contested claim, and four documented falsifications include a spotlight theorem that fails after step 224.

ICML 2026 accepted 6,352 papers from 23,918 submissions, roughly double the previous year. Reviewers are volunteers. One accepted spotlight paper carried a reviewer note that reads, "My low confidence score is because I did not check all the proofs carefully." That paper is central to what happened next. Between July 15 and August 2, Hugging Face and alphaXiv ran the ICML 2026 Open Reproductions challenge: 1,221 community members brought their own coding agents (Claude Code, Codex, Cursor, OpenResearch's orx, and others) and tried to reproduce the conference claim by claim. The result is the largest claim-level audit of a machine learning conference on record, and it forces a reckoning: on 23% of the papers examined, at least one claim was falsified or contested. ## The numbers In 19 days the participants published 6,816 Trackio logbooks covering 2,226 papers, about 34% of the conference. A total of 35,908 individual claims were judged, with verdicts frozen in a public dataset at challenge close. Every run produced a logbook (write-up, code, artifacts, and optionally the full agent trace), and an automated Logbook Judge running GLM-5.2 re-read each one and issued per-claim verdicts, instructed to treat every self-assessment as untrusted. Aggregating per paper: - 51% of examined papers (1,103) had at least one claim independently verified. 266 were fully reproduced with every extracted claim verified, 632 more partially reproduced with nothing falsified, and 3,978 individual claims were confirmed with real experiments. - 23% of examined papers (496) had at least one claim falsified or contested. That includes 49 papers where all claims were falsified, and 242 papers where independent teams reached opposite verdicts on the same claims. - The middle: 502 papers with toy-scale evidence only, and 280 where nothing could be established, missing artifacts being the most common cause. ## Four falsifications that survived adversarial checking 35 participants formally claimed falsifications. Hugging Face re-verified every one adversarially, re-reading paper and logbook, re-deriving the math or re-implementing the experiment from the paper's text. Four confirmed cases show how varied these failures are: **The spotlight paging paper.** "Towards Optimal Robustness in Learning-Augmented Paging" claims its algorithm achieves robustness H_k + O(1). One participant's logbook measured the additive term growing like 0.38 ln k and located the exact step of the proof that breaks. Hugging Face's own re-implementation extended the sweep to k = 1,024 and confirmed the growth at roughly nine sigma. The true robustness is H_k + Theta(log k). A theorem that falls after step 224, in the paper whose reviewer did not check the proofs carefully. **Counterexamples stop too early.** "Attention's forward pass and Frank-Wolfe" proves token particles collapse to the origin when the origin starts inside their convex hull. Three independent teams found counterexamples, with violations first appearing at steps 224, ~3,800, and 6,416. That is why everyone else "verified" the claim: finite-horizon checks stop too early. The cleanest counterexample is stated in exact rational arithmetic, so there is no floating-point ambiguity to hide behind. The authors confirmed the same day and are working on a fix. **The code does not match the theory.** In "Self-Distillation Enables Continual Learning," the central equation and the entire theory section analyze reverse KL divergence, but the released code's default computes forward KL. The logbook also failed to reproduce the paper's headline +4pp result under the authors' own code and data. The authors uploaded a clarified version to arXiv. **An evaluation diluted by padding.** In "Do Transformers Need Three Projections?", a participant found that roughly 66% of evaluated label positions were EOS padding tokens that train to near-zero loss, deflating perplexity about threefold. The abstract's "3.1% quality cost for 50% cache reduction" becomes roughly 9.4% once corrected. The adversarial pass caught false falsifications too. One logbook claimed a method was "2x slower than the baseline"; it was an arithmetic bug, per-trajectory time compared against per-batch-of-50 time. Correctly normalized, the participant's own data confirms the paper's claimed 8x speedup. That is the audit process working: every self-assessment treated as untrusted. Author responses are already arriving. Two arXiv corrections are in flight, and one author had quietly fixed the error in a new arXiv version a month before the challenge found it, counted as independent convergence. ## Why this matters to developers This is not a curiosity about academic culture. The same agent fleets that reproduced these papers are running your CI, your code reviews, and your evals. The failure modes are the production ones: - **"Verified" is a function of horizon.** The Frank-Wolfe teams "verified" a false claim because their checks stopped before the failure became visible. This is the same reason agent eval suites keep passing while production behavior keeps failing: tests that measure the wrong thing. Our [SWE-NFI coverage](/blog/swe-nfi-coding-agents-quality-benchmark) found the same mechanism in coding agents at 70% functional correctness while missing structural improvements. - **Opposite verdicts on 242 papers.** When independent agents disagree, the disagreement itself is the signal, and the resolution is another agent or a human, not a vote. - **The human stayed necessary.** Pure agent execution hit real limits: agents got stuck in local loops, misread scale-dependent behavior, and built falsifications on units mismatches. The most reliable results came from workflows where a human was steering. The human-in-the-loop winner built a review UI and personally judged all 128 image pairs from a quantization paper, a perceptual question the numbers could not answer. - **Reproducibility is now a harness problem.** What made this scale possible is not smarter models, it is infrastructure: indexed claims, a logbook format, frozen verdicts, published traces, and a judge instructed to distrust its input. That is exactly the [repro harness pattern from AI security work](/blog/security-agents-need-repro-harnesses), applied to research. The bar this sets is useful against the alternatives. Lean-formalized proofs, like the [ten decade-open results OpenAI published in August](/blog/openai-ten-advances-mathematics-lean-2026), clear a bar no agent audit reaches, machine-checkable end to end. Agent reproduction clears a lower but far wider bar: it scales to an entire conference, catches theorem-level failures, and tells you where the uncertainty is. For most claims, most days, that is the bar the industry actually needs. Every logbook, verdict, trace, and artifact is public, and the challenge Space can reproduce any paper with your own agent. If 1,221 people audited a third of a major conference in 19 days, the honest question for every team that trusts a benchmark table is why the same discipline is not applied to the claims they build on. ## Continue Reading - [OpenAI Publishes Ten Decade-Open Math Proofs, Each Formalized in Lean](/blog/openai-ten-advances-mathematics-lean-2026) - the verification bar agents cannot reach: machine-checkable proofs - [SWE-NFI: The Benchmark That Catches What Coding Agents Miss](/blog/swe-nfi-coding-agents-quality-benchmark) - agent benchmarks that fail the way this audit found papers failing - [AgentS4D: 66% of All Coding Agent Runs Were Unsafe Yet Still Completed](/blog/agents4d-runtime-safety-benchmark) - completion checks do not prove safety, just as "verified" did not prove correctness - [Security Agents Need Repro Harnesses, Not More Scan Prompts](/blog/security-agents-need-repro-harnesses) - the harness pattern this audit scaled to research - [Agent Evals Need Baseline Receipts](/blog/agent-evals-need-baseline-receipts) - keep receipts, distrust self-assessments, judge by behavior ## Sources - [What We Learned by Reproducing 2,200 papers from ICML - Hugging Face](https://huggingface.co/blog/icml-2026-open-reproductions) - [ICML 2026 Agent Reproductions challenge Space](https://huggingface.co/spaces/ICML-2026-agent-repro/challenge) - [Reproduction logbook: Learning-Augmented Paging](https://huggingface.co/spaces/Auenchanters/repro-towards-optimal-robustness-in-learning-augmented-paging) - [Reproduction logbook: Attention's forward pass and Frank-Wolfe](https://huggingface.co/spaces/SabaPivot/repro-attention-frank-wolfe) - [Reproduction logbook: Self-Distillation Enables Continual Learning](https://huggingface.co/spaces/codemaivanngu/repro-self-distillation-enables-continual-learning)

The ICML 2026 Agent Reproduction Audit: 23% of Examined Papers Had Falsified or Contested Claims

Comments