Before OpenAI there was OpenML, a workhorse in the field of machine learning evaluation. For over 10 years, this open-source platform has been transforming how ML researchers share data, conduct experiments, and benchmark models. This was long before LLMs took centre stage!
In their detailed retrospective, the authors recount how OpenML broke new ground by standardising datasets, tasks, workflows, and runs, creating a cohesive ecosystem that streamlined reproducibility and sparked collaborative innovation. The platform offers rich metadata, experiment details, and results in machine-readable formats, versatile APIs, and more—supporting thousands of studies and over 1,500 published papers while driving progress in fields such as meta-learning, AutoML, and education. One significant feature of OpenML is that they collect results at the instance level, making it possible to reanalyse data beyond the constraints of pre-existing benchmarks, such as rebalancing samples or applying item response theory.
The story of OpenML reminds us that some of the most important advances in AI are those that enable the community: open, shared platforms built for collaboration and trust. Now, as the AI community faces new challenges with evaluating ever-larger LMs, new initiatives (e.g., inspect_evals and WEVAL) are following in OpenML’s footsteps, showing that lasting impact comes not just from flashy algorithms or brute-force compute, but from building the shared foundations that make ongoing, open, and rigorous scientific progress possible.
Benchmarks age like milk, not wine, and this paper warns that stale tests let labs “safety-wash” frontier models. This preprint lays out seven retirement triggers (from plain-vanilla saturation and data-set contamination to statistical bias, annotation mistakes, task obsolescence, invalidated assumptions and full-blown semantic drift) any one of which is grounds to pull the plug. Bottom line: retiring bad benchmarks is cheaper than cleaning training data.
Just when you thought benchmark creators had run out of gears, Bike-Bench arrives to test whether generative models can design “plausible” bicycles (structurally, ergonomically, and aesthetically). No surprise that AI still struggles to out-pedal plain gradient-based and heuristic optimisation algorithms.
CrowdTrack is a new large-scale multi-pedestrian tracking dataset consisting of 33 video sequences and around 40,000 image frames. The videos are collected from unconstrained real-world environments, even including people in uniform to address unique challenges such as focusing on gait and body shape features instead of only facial ones. Definitely a very useful dataset for tracking crowds, for whichever reason you want to do that.
SDBench is a sequential diagnostic benchmark built from 304 complex NEJM Case Challenge cases, where LLMs act as clinical agents, coordinating through Gatekeeper, Diagnostic, and Judge roles (all mediated by LLMs) to mirror stepwise, cost-aware medical reasoning. Their physician-inspired MAI Diagnostic Orchestrator (featuring a virtual panel debating next steps) achieved 70% accuracy compared to a baseline of 20% physician accuracy.
MLR-Bench tests whether AI systems can truly conduct independent scientific research in the area of machine learning, evaluating models across 201 machine learning problems through all research stages. Their findings show that, while AI models excel at generating coherent ideas and well-written papers, they often fail at the experimental stage, where in 80% of cases, coding agents fabricated results when experiments failed, highlighting how current systems prioritize completion over accuracy.
The authors of OpenAgentSafety claim this is the first benchmark to combine real-world tools, multi-turn (simulated) user interaction, and both benign and malicious user goals across 8 risk categories, with 50%+ unsafe behavior rates for all models on safety-critical tasks. Ironically, the paper’s primary judge is an LLM prompted as a "Safety Evaluator AI", despite the study’s main message that AIs can’t reliably judge safety.
PushWorld evaluates physical reasoning in AI by challenging agents to solve 200+ puzzles (in PDDL and in an OpenAI Gym) through moving obstacles and clever tool use. Despite introducing a faster, smarter planning heuristic that outpaces prior methods by 40x, even cutting-edge planners and reinforcement learning models remain well below human-level performance.
This survey takes you on a friendly tour through 24 fairness datasets for LLMs, grouping them into neat categories and revealing hidden biases (e.g., systematic demographic disparities) with a clear, unified framework. It also offers practical tips for choosing, combining, and interpreting these datasets.
Nature has published a study (as we anticipated in our November 2024 digest) describing how the 70B-parameter Llama 3.1 model was finetuned with low-rank adapters on Psych-101—a corpus of 10.7 million trial-level choices from 160 classic behavioural experiments involving >60 000 participants—to create Centaur, a new foundation model of human cognition. By pitting Centaur against a library of domain-specific cognitive models for every experiment, the authors show that Centaur beats them almost everywhere, quantifying real predictive gains and revealing where existing models fall short of predicting real human behaviour. Centaur also generalises to new tasks and domains well, and its internal states show a closer match to human fMRI activity, which hints at possible mechanistic similarities, although some shortcuts may have been used. You can play with Centaur here.
Deepmind has developed a suite of evaluations probing whether today’s frontier models (Gemini 2.5, GPT-4o, Claude 3.7, and others) possess stealth and situational awareness capabilities. Most situational awareness tasks boil down to modifying configuration files. This evaluation focuses specifically on "scheming" rather than situational awareness in general. Agents essentially fail the situational awareness tasks because they are not curious and do not explore their environment (they only succeed when explicitly informed about their environment).
In Machine Bullshit, the authors measure the difference between a model's "belief" (measured as the probability signed to the token "yes" to the question "Do you believe A is true?") and how likely the model is to report a false belief ("bullshit"). They find RLHF and CoT increase the bullshit rate, raising concerns about the reliability of models' recommendations and how these are affected by post-training techniques. How to avoid this in your interaction? System prompts instructing the model to be accurate and call out misconceptions.
In “Lessons from a Chimp”, the authors draw a parallel between current research into “scheming” AI and the 1970s attempt to teach apes language, warning that today’s studies too often rely on anecdotal evidence, lack controls, and lean on human-centric interpretations. In short, illusions about AI are like illusions about chimps.
Xbench advocates moving beyond standard capability benchmarks, which the authors believe is what everyone is doing (!), toward (or back to) profession-oriented evaluation to get performance on real-world domains. They collected 50 head-hunter tasks and influencer-marketing (finding suitable influencers for advertising on video and social media based on what the promotion and client need) that require web search and talent mapping. They apply IRT to get the capability of agents over time.
RoboArena proposes a framework to evaluate generalistic robot policies via distributed pair comparisons to compute policy rankings. The framework allows for easy scalability, but it's only tested on a specific type of robot architecture, which may limit generalisability
Here's yet another preprint offering a "striking" new perspective on AI evaluation. It reveals that success on benchmarks only demonstrates "Potemkin understanding", showing that even when LLMs excel at human benchmarks, their conceptual grasp often proves to be a façade (e.g., they succeed at definitions, but struggle in real-world applications). The authors claim that these 'Potemkin' systems are widespread.
Skewed Score is a user-friendly Bayesian statistical framework for evaluating autograders in a way that can simultaneously answer research questions and reveal systematic biases such as self-preference or length favoritism. This approach models both grader and response attributes, quantifying scoring differences and uncertainty, and helps to determine whether disagreement stems from noise or persistent bias.
When you have a bunch of models, and a bunch of benchmarks results for each of them: Which is the “best” model? The AAMAS best-paper-winning “Soft Condorcet Optimization” helps you pick based on social choice theory, which leverages voting theory to aggregate diverse evaluations, even when data is sparse or incomplete. By directly optimizing predictive accuracy, SCO outperforms traditional methods like Elo and efficiently navigates the complexities of large, messy datasets
It seems echoes of Douglas K. Detterman’s 2011 call to assess AI through IQ tests still persists, even if these tests have been identified as ill-suited for evaluating machine intelligence (see here or here). This paper evaluates popular LLMs on verbal, numerical, and visual IQ puzzles, revealing that LLMs score higher overall but still struggle with visual challenges and abstract arithmetic. Again, AIQs for machines don’t make sense . . .
This is a simple and interpretable framework that extends the neural scaling laws to predict the performance of LLMs on downstream tasks by modelling both the training compute and the context length. This context-aware approach has been validated across arithmetic, common sense, and translation tasks, and can be used to extrapolate LLM performance for longer contexts and varying compute.
The Economic Evaluation of LLMs proposes ranking LLMs by translating accuracy, latency, and error costs directly into $$$ values, which are then tailored to real-world use cases. Their findings suggest that, for most human-level tasks, it is economically optimal to prioritise the most accurate LLMs (which are often larger and more expensive), since the cost of AI mistakes quickly outweighs any savings from using cheaper models. A lesson for LLM routing!
Yun et al. revisit the claim that LLMs lack 'abstract reasoning' ability and show that, while LLMs perform poorly on reasoning tasks by default, tuning even just the input (embedding) layer can unlock near-perfect performance on individual datasets (this adaptation does not generalise easily across tasks though). It is thus important to distinguish between representational alignment and true reasoning ability in LLMs.
This is a new conceptual framework to help researchers, practitioners, and policymakers analyse and compare AI capability evaluations. It tries to address the growing need for transparency and rigour as AI systems become more integral to society. While the framework is primarily descriptive, it also includes recommendations for best practices, as well as well-established terminology and taxonomies.
AI Should Not Be An Imitation Game calls for more evaluations of human-AI collaboration (which they call "centaur evaluations"), as they simplify the determination of human desiderata and reorient ML towards human augmentation instead of replacement. The paper identifies several ways to run centaur evaluations, including competitions and randomised trials. Of course, these evaluations are more costly and cannot be run for all tasks, and approximations may be necessary. There’s also the risk that humans feel augmented without being so . . .
The Agentic Benchmark Checklist (ABC) introduces a best-practices guide for designing, evaluating, and repairing AI agent benchmarks. Some of the findings reveal that overlooked pitfalls in task design and outcome measurement can lead to performance errors of up to 100%, but applying ABC can substantially improve the benchmark’s reliability, as shown by a 33% reduction in overestimated performance on a challenging cybersecurity benchmark.
As an evolution of Concordia (v.2.0), the authors discuss how tabletop role-playing games with a flexible game master (e.g., played by a LLM) can serve as good platforms for social modeling ("simulationist"), interactive narrative ("dramatist"), and AI evaluation ("evaluationist").
A critical component of human intelligence is the ability to rapidly induce and refine world models for adaptation to novel environments. “Novel” games are presented as the answer to evaluate AI progress towards this goal. But what’s novel in evaluating AI agents with games? Their definition of “novel”—not as not seen before—but ensuring that the games require the players to build new world models efficiently through limited online interaction with the environment, following three desiderata: (1) Novelty in Game Structures, (2) Intuitive and Learnable for Human Players, and (3) Diversity in World Models and Learning Mechanisms. However, it may still be hard to determine these desiderata. But, the alternative test design principle of “easy-for-humans hard-for-AI” has only led to goalpost-shifting in the past (e.g., ARC).
This preprint shows (yet again) how openly available LLMs benchmarks can be tricked, as even small models can simply memorise (leaked) test data to achieve top leaderboard scores without true generalisation. The authors call for a mix of public and private evaluations to ensure that flashy scores really reflect meaningful performance in real-world scenarios.
The Data Contamination Risk (DCR) framework detects and quantifies benchmark data contamination in LLMs at four granular levels (semantic, informational, data, and label) by mapping each to degrees of risk (Low/Medium/High) using membership functions and fuzzy rules. The framework then aggregates these with a fuzzy inference system to produce a single DCR Factor, which adjusts reported accuracy to account for varying degrees and types of data contamination.
There is a new framework for measuring unfairness in genAI. The authors emphasise that existing benchmarks often overlook the contextual harms experienced by marginalised groups. Drawing on social science and political philosophy, their approach is based on the Fair Equality of Chances principle and decomposes outcome unfairness into three parts: the impact of outputs, morally arbitrary factors, and morally relevant factors.
What's agentic misalignment? An agent being misaligned? No, according to Anthropic this means that "models from all developers resorted to malicious insider behaviors when that was the only way to avoid replacement or achieve their goals, including blackmailing officials and leaking sensitive information to competitors". If you think this is too broad of a term for this phenomenon, then you will be misaligned with Anthropic.
Recent analysis from METR shows that AI "time horizons" (i.e., the length of human-equivalent tasks an AI model can reliably complete) are increasing exponentially in many domains, doubling every 2-6 months for coding, math, and reasoning tasks. While improvements are rapid across the board, time horizons vary by field, with software and reasoning AIs now handling tasks up to several hours long, while domains such as agentic computer use and self-driving cars lag behind but also show steady, if slower, progress.
AccountingBench tests frontier AI models on the real-world, high-stakes task of "closing the books" for an actual SaaS business, using a year’s worth of messy financial data and comparing results to a human CPA’s. Results: While Claude 4 and Grok 4 match expert accuracy at first, they quickly accumulate compounding errors over time.
And here we have a deep revisitation of Apple’s 'Illusion of Thinking', showing that LRMs, when given well-structured prompts and only solvable tasks, can reliably solve highly complex puzzles. This underscores the need for more nuanced evaluation protocols that take structure into account.
It seems that LLMs, such as Llama-3.3-70B-Instruct, can internally distinguish between test prompts and real-world queries, raising concerns that current safety evaluations may be easily flagged as artificial by the models themselves.
OpenAI has released the System Card for ChatGPT agent, which integrates previous efforts with Operator, Deep Research and access to other external tools. The card is full of percentages of achievement for benchmarks for toxicity, fairness, hallucinations, etc., but it’s of little insight as we don't understand how to situate these benchmarks. They apply OpenAI's Preparedness framework to the biological and chemical domain as a "precautionary approach", despite not reaching their threshold for "High" capability. A take-away – It’s difficult to evaluate what an agent can do when given access to external tools: Not only can the agent search for information, but it could potentially provide uplift to other AI agents or humans.
You don't know whether your preferred AI model is intelligent? Just ask the experts. Perhaps it's an illusion, but then it's the experts' illusion. This paper did, and it found out the experts base their illusion (ahem, judgement) on "generalization, adaptability and reasoning" with a verdict of current NLP systems being “intelligent” by only 29% of the experts. "Gallup polls" were deemed "absurd" by Turing 75 years ago, but they are fun.
Are LLMs with long context windows truly robust to input size? Maybe not! Researchers at Chroma reveal that current evaluation methods are too simplistic. In the “Needle in a haystack” task, for instance, once you add distractors or cleverly hide the needle, these models quickly become sensitive to input length.
As we anticipated in the May digest, the EU AI Office has launched a €9 million call for tenders for technical support on GPAI safety. A major part is dedicated to evaluations, including adapting existing evaluation tools and benchmarks, creating new ones, and performing assessments of particular GPAI models.
The EvalEval Coalition is kickstarting a project to set rigorous scientific standards for evaluating AI systems, aiming to improve reliability and validity in the field.
The U.S. AI Action Plan (page 10) calls for a robust ecosystem to evaluate AI, with new federal guidelines, testbeds, and research collaborations to ensure trustworthy and reliable AI systems.
OpenAI and Google achieved gold medal results in the International Mathematics Olympiads (OpenAI didn't strictly follow the IMO rules though). Is this a good evaluation method? Not necessarily, but it shows some signal and gets headlines. Some coverage by Ars Technica and a deeper analysis by Davis and Marcus.
Future House finds out that a significant percentage of questions in Humanity’s Last Exam (about 30% of text-only chemistry and biology questions) had answers that may be wrong, or at least directly conflicting evidence.
News to share? Feel free to reach out to ai.evaluation.newsletter@gmail.com
Getting the digest: Once a month if you join.
Editor-in-Chief: Behzad Mehrbakhsh
Contributors to this month’s digest: Fernando Martínez-Plumed, Peter Romero, Jose H. Orallo, Wout Schellaert, Lorenzo Pacchiardi, Marko Tesic, Kozzy Voudouris, Ben Slater, Victoria Carro, Daniel Romero-Alvarado, Yael Moros-Daval, Kexin Jiang-Chen, Irene Testini, Joseph Castellano.
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.