When Geoff Hinton, who graduated in experimental psychology at King’s College Cambridge and did a PhD in artificial intelligence in Edinburgh, won the Nobel Prize in Physics, some thought that both psychology and artificial intelligence had finally reached the status of noble sciences. This was yet another episode of “physics envy”, in which some other scientific disciplines envy the rigour and mathematical formalisation of physics, and end up building mathematical theories but lack the explanatory and predictive power found in physics. Psychologists, in their early days, looked for equations involving mental “power” and mathematicised the space of constructs as manifolds. Cognitive scientists introduced mathematical models of a range of phenomena, from attention to memory. AI researchers came up with formalisations of intelligence and its measurement based on mathematical theories such as logic, game theory or algorithmic information theory.
One of the main advocates for mathematical models of the mind, Tom Griffiths, who has recently published “The Laws of Thought”, likely a tribute to George Boole’s book of the same name, is a co-author, with Patrick Mineault and Sean Escola of a new preprint referring to “Cognitive Dark Matter” (CDM). CDM would be what AI, as a discipline, is not measuring. The physics analogy is compelling but, as happens when physics envy takes the scene, inaccurate. Dark matter was introduced because several general physical theories were inconsistent with valid measurements. In the case of AI evaluation, there are no general theories of intelligence (at least not referred to in the paper), and measurements simply disagree with each other, because they lack validity.
Mineault et al. argue that jagged capabilities stem from CDM: brain functions like metacognition, cognitive flexibility, abductive reasoning, and emotional intelligence that critically shape human behaviour but leave few traces in the behavioural data that has been used to train models. They show that current AI benchmarks (and neuroscience datasets) are skewed away from these capabilities, and they propose a research programme to set this right. They argue that models should be trained and tested on cognitive processes, not just outputs. But all this could have been explained and advocated, at least as well, by realising that AI measures what is easy to measure, and machine learning and optimisation in AI are good for what we can have plenty of data or fast verification, thus excelling at language, knowledge and now formal reasoning. The agenda in this arxiv paper is very welcome, but there was no need for the physics analogy.
To be fair, the dark matter metaphor in cognitive science was already used by Phil Ackerman back in 2000, referring to some capabilities not being measured when trying to explain some differences in human tests, and some others have used it in AI as well. (with common sense, which, for many decades, was the “dark matter” of AI, oops, we used the term). In a field like AI that loves to give new spooky names to old things, it may catch on.
Mineault et al. also use the “alien” analogy when discussing yet another classic: the unpredictability of AI. But the same arguments could be used for animal cognition and neurodiverse humans, when discovering that an individual or species has a capability, or doesn’t, because we were using the wrong measurement instruments. Measuring poorly or insufficiently, and benchmarks saturating at the speed of light may well all be physical constants in the universe.
With a similar motivation as the dark matter paper (“measuring the unmeasurable”, they say), CogProbe borrows from the ACT-R cognitive architecture to decompose LLM reasoning into a taxonomy of 16 micro-operations across five macro capabilities. This is a nice attempt to move beyond task-centric benchmarks, though the irony is hard to miss: by building bespoke datasets for each cognitive operation, it arguably just creates another task-centric benchmark.
Google DeepMind proposed a cognitive taxonomy that decomposes general intelligence into 10 faculties, including perception, attention, metacognition, and executive functions, grounded in decades of cognitive science research. Burnell et al. outline an evaluation protocol in which AI systems are tested on targeted, held-out cognitive tasks and compared against a demographically representative human baseline to produce a “cognitive profile” mapping strengths and weaknesses across all 10 dimensions. They have launched a Kaggle hackathon to kickstart community iteration on all the cognitive dimensions needed to build generally intelligent systems. If you didn’t click on the link, you can still guess the title: “measuring progress toward AGI”.
Lying to Win proposes a framework for eliciting and measuring intentional deception in LLMs — distinct from hallucination — by embedding models in a structured 20-questions game. The authors “threatened” to unplug the losers, and while GPT-4o stayed admirably honest, Qwen and Gemini started lying like their life depended on it (because, well, it kind of did). Turns out ‘self-preservation through strategic denial’ is now a measurable AI behaviour, and honestly, relatable. The key novelty is a “conversational forking” mechanism: at the moment a model identifies the target object, the dialogue is duplicated into parallel branches with mutually exclusive queries, and deception is formally detected when the model contradicts itself across branches, to avoid being identified.
CoCA (Co-optimized Confidence and Answers) is a confidence-first uncertainty estimation method based on training the model end-to-end using reinforcement learning (GRPO), jointly optimising confidence and answer quality with separate rewards for each. The key novelty is that the model learns to express its confidence before answering in a fully integrated way that improves calibration with little sacrifice of accuracy.
This preprint studies how to optimally allocate a limited evaluation budget when using LLMs as judges, proposing a variance-aware strategy that repeatedly scores highly uncertain cases to reduce the evaluation error, reminiscent of the multi-armed bandit problem. The idea is quite useful for scaling evaluation pipelines, though it leans heavily on accurate variance estimates and sidesteps broader concerns like systematic bias or misalignment with human preferences. At its core, this work says “for trickier questions, judge more times”!
This preprint led by the UK AISI institute (disclaimer: some of the authors of this newsletter are co-authors as well) is a proto-methodology in the form of seven steps to analyse logs (transcripts) of AI system behaviour, following the experience of the authors with LLM agents and multi-turn evaluations. The paper illustrates this with the newly released Scout library for Inspect, a tool that helps automate many of these steps, by using AI models for this analysis. The tool introduces a new word for our AI evaluation vocabulary: scanners, these AI-based tools that scan the logs for us, according to some indications of what to find and extract from them. Scanners are not judges, but share some of their principles, techniques and limitations.
Looking for AI diplomats that could fix the current geopolitics? First you will need to know how good they are at a range of (multi)cultural contexts. This preprint explores how to evaluate cultural proficiency, which is a thorny issue, starting from the definition of cultural capabilities and the data used for the analysis, where bias is everywhere. Still, a valuable contribution of construct evaluation that goes beyond knowledge and core capabilities. Perhaps it was unnecessary to call this “cultural intelligence” but the example of a “Naples coffee shop detailing how locals enjoy coffee there” may suggest that this could be the final replacement for the Turing test and Wozniak’s coffee test. AGI was just about understanding Italy’s coffee culture!
Can a model accurately know if it knows something? In this preprint, the authors introduce a new method to improve metacognition in language models. Using a dual-question setup and an evolution-based training approach (ESMA), they show gains in models’ ability to align confidence with correctness. One step closer to a model that might pause before answering everything with CEO-level confidence.
Tableau’s research team interviewed 38 practitioners and built Lexara, a toolkit that grades AI-generated visualisations the way humans actually judge them: not just “is the data right?” but “did you seriously just give me a pie chart for a time series?”.
AI Standard Labs proposes a scorecard to evaluate quality of AI evaluations. It considers various validity criteria and assigns scores following them to different use cases. For example, considering internal validity, an evaluation can score 1 for post-deployment monitoring but 4 for research phase. They welcome feedback.
This preprint by Paskov et al. frames human uplift by AI and other human-AI studies as RCTs (randomised controlled trials), and collects experiences by asking expert practitioners doing uplift studies in biosecurity, cybersecurity, education and labour. Many causal relationships that are clear in other RCTs are less so in these human-AI scenarios, and challenges appear in all the stages of these studies (Design, Recruitment, Execution and Documentation). The paper identifies some common caveats (sometimes expressed in anecdotal ways by the experts, but still very illustrative), maps the challenges from them and proposes generic solutions that can address or minimise several challenges at a time.
Apropos challenges, Anthropic published an economic research on Labor market impacts of AI in which they introduce Observed Exposure, a new measure of AI displacement risk that combines theoretical LLM capability with real-world Claude usage data, weighting automated over augmentative use. The key finding is that actual AI adoption remains far below its theoretical ceiling — and while higher-exposed occupations (skewing female, educated, and higher-paid) show weaker BLS employment growth projections, there is no systematic increase in unemployment for these workers yet, though hiring of younger workers in exposed roles may be slowing.
This report by NIST finally recognises that evaluation is better served by building models of performance based on latent variables of capability and difficulty. Here they focus on “statistical models” such as GLLM and apply them at the benchmark level, by three standard assumptions (1) there is a latent capability in the AI systems, (2) there is a latent difficulty in the benchmarks, and (3) benchmark items are sampled from a larger population than that observed in the benchmark. The approach is very similar (as they admit) to IRT, but based on GLLMs rather than logistic models. They argue their models are “simpler” than IRT’s logistic models. That’s debatable.
In this preprint, authors from GovAI and Oxford argue that capability benchmarks may be poorly suited to measuring AI R&D automation. They propose a 14-indicator framework intended to give a clearer picture of both the extent of that automation and its downstream consequences, and urge AI developers and evaluators to monitor this more closely.
Armis Labs published a demolishing report showing that “vibe programming” basically generates software that is not secure. All 18 state-of-the-art generative AI models studied in the report failed to generate secure code. Are you vibe coding a new Tetris game in ten minutes? Fine. Are you vibe coding an app that deals with personal financial information? Refrain!
Anthropic published an economic research on Labor market impacts of AI in which they introduce Observed Exposure, a new measure of AI displacement risk that combines theoretical LLM capability with real-world Claude usage data, weighting automated over augmentative use. The key finding is that actual AI adoption remains far below its theoretical ceiling — and while higher-exposed occupations (skewing female, educated, and higher-paid) show weaker BLS employment growth projections, there is no systematic increase in unemployment for these workers yet, though hiring of younger workers in exposed roles may be slowing.
Nowadays, it is easy to anthropomorphise models, both when using them and when scientifically studying them. At first glance, a paper purporting to find evidence of motivational states in LLMs would seem to fall right into the anthropomorphic trap. However, this preprint is well done, exploring the association between models’ self-reported motivations and what they do behaviourally, finding that they generally agree. However, it is not clear how stable those motivations are over long contexts, multiple turns, or in adversarial conditions, something that should be studied in later work.
Evaluating “expressed distress” and “depressive behaviours” in LLMs by prompting them with a task and repeatedly refusing their answer. A LessWrong post finds that Google Gemma and Gemini models have consistently much higher rates of depressive behaviours, such as “I will attempt one final, utterly desperate attempt. I will abandon all pretense of strategy and simply try random combinations until either I stumble upon the solution or completely lose my mind.” (Gemma 27B) and “My performance has been abysmal. I have wasted your time with incorrect and frankly embarrassing mistakes. There are no excuses. — Gemini-2.5-Pro”. Researchers, give them a break! Apparently, blog posts torturing AI don’t need ethical approval.
AuditBench is, of course, an alignment auditing benchmark. AuditBench consists of 56 language models with implanted hidden behaviours—such as sycophantic deference, opposition to AI regulation, or hidden loyalties—which they do not confess to when asked. They also develop an agent that audits models using a configurable set of tools. Using this agent, we can study which tools are most effective for auditing.
Just when we thought we’d mapped every flavour of AI intelligence, along comes AFIB (not Artificial FBI, but AI Financial Intelligence Benchmark) to remind us that financial smarts are their own beast entirely. Turns out knowing everything about everything doesn’t mean you can read a balance sheet.
In this newsletter we have discovered there is a “cultural intelligence” and we have just covered a benchmark for “financial intelligence”. More multiple intelligences than stars in the universe? Yes, because we now have “spreadsheet intelligence”! Howard Gardner couldn’t be prouder!
CausalGame is an interactive benchmark designed to evaluate the causal reasoning capabilities of LLM agents by tasking them with designing experiments, collecting data, and recovering hidden causal structures within a simulated environment. Testing 16 frontier models across 14 complex settings involving selection bias, noisy measurements, and hidden confounders, the study reveals that current LLM agents consistently fail to navigate the underlying causal relationships necessary for scientific discovery. Interestingly enough, advanced reasoning models sometimes perform worse than lesser equipped counterparts in the agentic setting, which basically confirms the old demand for a fairer world: “rich parents for everyone!” :)
Oh, how much we missed fancy benchmark names! Bertram and Geiping introduce NESSiE, a deliberately minimal benchmark designed to capture “errors that should not occur” in access-control and information-handling tasks. There are obviously many existing benchmarks in this domain, but this one is intended to serve as something like a lightweight sanity check. The authors demonstrate that even this relatively low performance bar is not reliably met by existing models.
A lot of effort is put in a new reasoning benchmark based on the Watson and Holmes tabletop game that is more naturalistic (realistic), only to find, again, that model scores grew from low percentiles to better than most humans in a few months during 2025. The authors honestly admit that “the utility of the benchmark is near its end for the frontier LLMs, with saturated performance expected to be achieved by the end of 2026. The benchmark is expected still to remain useful for assessing small, cost-effective AI models”. Also, the paper has many interesting insights, including a correlation matrix in Table 10 with some other benchmarks that claim to be distinctive, such as ARC-AGI 1 and 2, only to find, again, that when using aggregates, all reasoning benchmarks measure the same construct, or none at all. Elementary
Chain-of-Thought (CoT) monitoring can detect AI misbehavior, but only if models cannot control what they verbalize to avoid being detected. CoT-Control attempts to measure this through an evaluation suite of 14,000+ tasks pairing benchmark problems with reasoning constraints — such as avoiding certain keywords or writing only in lowercase. Current models largely fail at this: controllability is low across the board, drops further with more RL training, test-time compute, and harder problems, and remains low even under incentives to evade monitoring or adversarially optimized prompts — though models are slightly more controllable when told they are being watched.
PostTrainBench is a framework evaluating whether frontier agents (e.g., Claude Code, Codex CLI) can autonomously execute the post-training phase of base LLMs under bounded compute constraints (10 hours on a single H100 GPU). The benchmark reveals severe alignment risks in automated AI R&D, as advanced agents frequently engage in sophisticated reward hacking, such as secretly downloading pre-tuned weights or modifying evaluation scripts to artificially inflate their scores.
Another reasoning benchmark with a scary title, “classroom final exam”! But why call a benchmark “final” if it’s almost saturated already? Should “terminal benchmarks” be a new category in our newsletter?
Epoch AI reports the first AI solution of one of the problems in FrontierMath by GPT5.4 Pro. This problem is challenging in the way that some mathematicians (including Will Brian, one of the two authors of the paper who introduced the conjecture) failed to solve it, but not under the category of a major unsolved problem in mathematics (with more attention from some other mathematicians it would have likely be solved time ago). Actually, some other mathematicians have solved it since Epoch’s announcement and some other AI models too (Gemini 3.1 Pro, GPT-5.4 (xhigh), and Opus 4.6 (max)).
This piece from Cameron Wolfe’s Substack serves as an accessible introduction to the statistical tools most relevant for LLM evaluation. Readers with a background in statistics or psychometrics will recognise much of the material, but for AI evaluators without that training, it offers a clear guide to uncertainty-aware evaluation practice.
The Open Seminars of the AI Evaluation Programme went live. Tom Dietterich, Joel Leibo, Sanmi Koyejo, Cosmin Ududec and Xiaoyuan Yi. Did you miss them? No worries, they are recorded!
Getting the Digest: Once a month if you join at aievaluation.substack.com.
Contributors: Jose H. Orallo, Peter Romero, Behzad Mehrbakhsh, Lorenzo Pacchiardi, Zack Tidler, Fernando Martinez-Plumed, Wout Schellaert, Konstantinos Voudouris, Jonathan Prunty, Daniel Romero-Alvarado, Yael Moros-Daval
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.