AI evaluation is not only benchmarking and red-teaming. More broadly, it can be defined as measuring and anticipating the behavioural properties of AI systems [...] to inform decisions about their use. As a consequence, studies that carefully identify and isolate a specific phenomenon are highly-valuable contributions to AI evaluation.
A great example of these studies is a work, published in Nature, that analyses the phenomenon of “emergent misalignment” (first observed in an Arxiv preprint by the same authors in February 2025). Basically, emergent misalignment happens when a model is trained to be misaligned on narrow tasks but this leads to broader misalignment. As the accompanying Nature commentary by Richard Ngo argues, the concept of emergent misalignment is useful to pin down a wide range of anecdotal observations (such as Grok or Bing Chat going rogue and jailbreaks prompting the model to take on alternative personas) that, when considered individually, seem surprising. Ngo argues that introducing new concepts for AI cognitive phenomena (such as “emergent misalignment”) is analogous to the transition of ethology from purely behaviourist descriptions to attributing mental traits to animals. As ethologists use these mental traits to better explain how animals behave in the wild, AI scientists can draw from the growing vocabulary of “AI cognitive processes and traits” to form predictions and explanations.
Indeed, even if the experiments of the original paper where emergent misalignment is clearly visible involve tricks such as fine-tuning, subsequent research by Anthropic found that RL training where reward hacking is possible can lead to “natural” emergent misalignment (i.e., without being explicitly trained to be narrowly misaligned). A concise and clear (and thus easily actionable) explanation of this would not be possible without the concepts of emergent misalignment and reward hacking in the first place.
Adding new concepts by which phenomena can be explained and predicted is part of the paradigm formation process of a scientific discipline. Works like this are manifestations of this process for the emerging science of AI understanding, thanks to great AI evaluation works.
Melanie Mitchell argues that current AI benchmarks systematically overestimate capabilities due to data contamination and the exploitation of spurious shortcuts, urging researchers to frame LLMs as “alien intelligences” rather than approximate humans. She proposes adopting rigorous experimental methodologies from developmental psychology and animal cognition—specifically perturbation tests and strict control conditions—to distinguish true robust reasoning from brittle, surface-level pattern matching. This perspective aims to shift evaluation from static leaderboard optimisation to a scientific inquiry that maps the precise boundaries of machine competence. It might be heretical, but as much as we want to see babies or animals as “alien,” they belong to the same biological classification system as we do, so the results might not be overly surprising. Also, good-luck testing all causal connections—we would have liked her to discuss the advantages of “white boxes” that AI systems have become, a tad more... but alas, that is cherry-picking on an otherwise great contribution.
Averi.org is a new organisation launched in January with the aim of making effective AI auditing universal. With the launch they presented a 114-page vision from 40+ coauthors (disclaimer: two of them collaborating with this newsletter), setting the agenda for frontier AI auditing based on four levels of assurance. The challenges are big, especially for the two higher levels of assurance.
To no one’s surprise, but for the first time at such a high level: Yann LeCun confirmed that Llama 4 benchmark results were “fudged a little bit.” While the community has long suspected benchmark manipulation, having a departing senior figure state it publicly marks a significant moment for evaluation integrity.
What is now generally referred to as “the METR graph” shows the ability of different LLMs in coding by measuring how long a set of tasks take humans and estimating the (human) length where each LLM achieves 50% correctness. This has an IRT feel to it, and we previously praised it for going beyond aggregate accuracy, despite the simple survival analysis interpretation. A new critique published on Transformer points out how tasks are unrealistic in some aspects, some may be contaminated, and the way to estimate human completion time is flawed. The critique argues these issues lead to a distributional shift in the estimated values, which may affect the widely-used “doubling time” of coding tasks LLMs can do. As a demonstration of intellectual honesty, one of the main authors published a blog post clarifying limitations and agreeing with some points, while still defending the doubling time estimate. While it is fair to maintain a high bar for such high-profile work, and METR claims to be working on improved follow-ups, we believe the original METR graph, despite being flawed, has been exceptional in building on a metric, time, that is predictive of success (in aggregation, variance across tasks of the same horizon length is high). Their results may be noisy estimates and should not be taken as set in stone, but they are at least a useful indicator.
Two preprints think that adaptive testing can minimise the number of evaluated items when trying to rank several AI systems, choosing the item (or benchmark) and the system that provides the most ranking information as evidence accumulates. One preprint derives a new IRT model focusing on continuous bounded scores (ROUGE, BLEU, LLM-as-a-Judge), while the other reframes benchmarking as active evaluation, exploring Elo rating and Soft Condorcet Optimisation. Elo is solid for reducing ranking error, yet SCO can rank agents better on Atari when tasks vary a lot.
A preprint introduces Competency Gaps (CG), an automated method that uses sparse autoencoders to decompose benchmark scores into fine-grained, interpretable concepts, revealing where LLMs and benchmarks systematically underperform. Applying CG to popular models and benchmarks shows that aggregated scores often hide consistent weaknesses, especially in safety, boundary-setting, and non-sycophantic behaviours.
Benchmark² essentially applies psychometric validation principles (the kind used to assess whether a mental test actually measures what it purports to) to LLM benchmarks, proposing three metrics that parallel classical concepts like convergent validity and item discrimination. Applying this framework to 15 widely-used benchmarks, they report substantial quality variation and suggest that you can reliably rank models using only about a third of the test items if you filter for quality.
In a preprint, Liu et al. argue that static multi-benchmark averages create a trap where peak performance in one area can overshadow total failure in another. Their “Swiss-System” tournament rewards “robust-generalist” models with all-around reliability while penalising “aggressive-specialist” models whose critical flaws are exposed under sequential task pressure—a topic that mirrors the classic psychometrics dilemma of “compensatory” vs. “multiple-hurdles” evaluation.
How many human judgments are enough? Lee provides a wake-up call for anyone interpreting human preference evaluations: nearly 1 in 5 model comparisons would need 500+ judgments to detect a winner, far exceeding what is typically collected. Many “inconclusive” results are not evidence that two models are equivalent—they are evidence that the evaluation was underpowered from the start.
A paper presents a lightweight LLM+RAG-based evaluation framework that scores scientific revisions and distinguishes venue-specific writing styles by basing judgments on retrieved “gold standard” conference texts. Its ratings accurately reflect real improvements in arXiv revisions and correctly identify NeurIPS, ICLR, and ICML stylistic signatures.
External evaluators need “access” to a model to evaluate it. The EU AI Act and its Code of Practice for General-Purpose AI says that access must be “appropriate.” To actually try to fill that in, a paper proposes a taxonomy of access methods for dangerous capability evaluation, and three corresponding access levels that should support cleaner communication.
A preprint shows that contaminating training data with test examples can dramatically inflate generative performance, similar to previously studied discriminative evaluations—but the effect is fragile. Crank up the temperature, and the memorised “germs” die off. The same way that heat sterilises a dirty surface, high sampling temperature effectively decontaminates evaluation performance.
A preprint on weird generalisation and inductive backdoors presents new ways to corrupt LLMs, showing how training-time manipulations can produce surprising out-of-distribution behaviours that standard safety evaluations fail to catch.
Do those “Wait, let me reconsider” moments in reasoning models actually help? d’Aliberti and Ribeiro examine whether mid-reasoning pivots in language models—sometimes held up as evidence of emergent self-correction—actually improve performance. Analysing a large corpus of reasoning traces, they report that such pivots are uncommon and correlate with lower accuracy, though artificially triggering reconsideration on uncertain problems did yield improvements.
Lin et al. tested autonomous agents against professional penetration testers in a live, 8,000-host environment. Aided by the authors’ ARTEMIS scaffolding framework, AI “teams” systematically outperformed most humans in speed and scale, proving particularly effective at exploiting legacy systems humans often overlook. While these agents still lack visual intuition and are more prone to false alarms, their ability to run massive operations at a fraction of human costs suggests that the next frontier of cyber-risk is as much about agent organisation as model intelligence.
Reitsam et al. introduce Pathology’s Last Exam, featuring text-based pathology cases designed to mimic real-world diagnostic problems, along with adversarial cases with biologically impossible information where the correct response is to refuse to diagnose. While some models performed reasonably on the diagnostic cases, they frequently offered confident diagnoses on the impossible ones—a safety concern worth flagging.
BabyVision curates 20 vision-centric tasks and evaluates models alongside children aged 3–12. The results are striking: even the strongest models perform at approximately the level of a three-year-old child.
A preprint investigates whether Vision Language Models share human-like cognitive constraints on inductive inference by replicating a classic developmental study that distinguishes between generic (“Bears”), universal (“all bears”), and indefinite (“some bears”) statements. VLMs align with human behaviour, extending properties in the hierarchy of all > generics > some. Post-hoc analysis suggests this stems from learned inductive constraints rather than surface-level pattern matching—though statistical mimicry, dependence on psycholinguistic nonsense words, and heavy text-reliance make us wonder whether other explanations exist.
GuardEval is a large, multi-perspective benchmark for evaluating LLM content moderators, accompanied by GGuard, a Gemma-based moderation model trained on it. Covering 106 fine-grained safety categories, the authors show that diverse, human-centred data significantly improve fairness, robustness, and performance over existing moderation systems.
Pluralistic alignment has been a natural reaction to monolithic versions of alignment. PluriHarms is a new benchmark that “colours” harms with the level of agreement of human populations and personas, also exploring the effect of prompting on that subjective experience of what may be potentially harmful. The paper attempts to predict this subjective perspective from prompts and personas, with mixed results.
T³ is a 454-vignette benchmark for causal reasoning that separates helpful endorsement from calibrated refusal across Pearl’s ladder, and shows safety-tuned models can over-refuse or hedge into counterfactual paralysis.
DAComp is two benchmarks in one: Data Engineering tasks evaluated through execution-based criteria, and Data Analysis tasks evaluated through LLM judges. For the latter, they evaluate six dimensions (Completeness, Accuracy, Insightfulness, Readability, Analytical Depth, and Visualisation) and split these into task-specific sub-requirements, with multiple “paths” corresponding to different ways an agent can solve the task.
TakeOverBench collects and groups benchmarks relevant to different model threats with a sleek visualisation. While it is a useful resource, they stress that “benchmark scores may not perfectly reflect the actual associated dangerous capability.” Also, their 1-year projections obtained with “automated mathematical modelling” are, to say the least, tentative and without confidence bounds. Valuable as an awareness-raising device, but limited as a decision tool.
A new accounting-focused benchmark (with “Multi-Calculation” set and CPA-style questions) finds that even top LLMs can follow multi-step calculations (~91% accuracy), yet still struggle with real accounting reasoning (~16–22%). Fluent answers are not enough.
News to share? Feel free to reach out to ai.evaluation.newsletter@gmail.com.
Getting the Digest: Once a month if you join.
Contributors: Lorenzo Pacchiardi, Jose H. Orallo, Behzad Mehrbakhsh, Fernando Martinez Plumed, Wout Schellaert, Konstantinos Voudouris, Peter Romero, Zack Tidler, Joseph Castellano.
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.