Every few weeks a new open-weight model lands with a headline benchmark score and a wave of enthusiastic posts. If you run models locally, the question is always the same: is this one worth downloading? The public leaderboards can’t answer that. They don’t know your hardware, they don’t test the quantised version you’ll actually run, and their test questions have a habit of leaking into training data, which means a model can be good at the benchmark without being good at the job.
So I built my own. More accurately, Claude and I built it: I set the requirements, made the calls, and checked the results; Claude wrote most of the harness code. For an article about measuring AI models, that division of labour felt appropriate. The result is not a grand research project, just a small, sharp harness I can point at any model that runs on my llama.cpp server, old or new, and get an objective, comparable score in about an hour. The goal was never to crown a universal champion. It was to have a repeatable method: when a model is released, run it through the same gauntlet as everything before it, and know where it sits on the three axes that matter for local use: how capable it is, how fast it runs, and how much memory it demands.
Here’s what the harness looks like, what it found, and the finding that surprised me most: a single benchmark run is not a measurement.
The core is five self-contained coding problems in four languages: a duration-string parser in Python, an LRU cache in JavaScript, SQL analytics, a concurrent worker pool in Go, and a scheduling algorithm. Each has an automated grader, 68 checks in total. There is no “LLM judge” grading essays about vibes; a check passes or it doesn’t. The graders and their hidden edge cases stay private, because a benchmark whose answers are searchable is a benchmark whose scores you can no longer trust.
Each model runs the gauntle ft three ways. The one-shot track gives it a problem and takes its first answer, no retries: raw code generation. The agent track makes it write the files itself through tool calls, the way a coding assistant actually works. And a reasoning track of 102 mechanically graded items (state tracking, constraint satisfaction, instruction compliance and more) probes general fitness rather than coding. Crucially, most reasoning items are generated from a seed and solved programmatically, so they cannot exist in any training set. Suspect contamination? Change the seed and get a fresh exam.
Alongside all of that, a speed sweep measures tokens per second, because a brilliant model you can’t bear to wait for is a model you won’t use.
Six models made it through every track, all quantised, all on the same machine. Getting there was not smooth: Kimi-Linear-48B-A3B never survived loading at all, and three of the six failed their first load attempt. Local inference is still a contact sport. The cross-track picture:
gpt-oss-120b wins every track outright: perfect scores on coding, agent work, and reasoning, with the most disciplined tool use I saw. It finished agent problems in under three turns on average, no wasted calls. One asterisk, and my own rules demand I state it: gpt-oss applies its reasoning regardless of the thinking switch, so its scores are with thinking on while everyone else’s are without. It still wins, but that gap is not measured on identical settings. If you can afford 63 GB of memory and 54 tokens per second is fast enough, nothing else has an argument.
But the interesting result is the value pick. gemma-4-26B-A4B delivers a perfect agent score at 74 tokens per second in just 14 GB, a quarter of the memory of the leader, running faster. The catch is that it is chatty: 8.6 turns per problem to gpt-oss’s 2.8, and it ran into the turn limit on three of five problems, so its wins cost roughly four times the tool calls. If your interest is a practical local coding assistant rather than a trophy, it is still the sweet spot on my hardware, just budget for the extra calls.
And gemma-4-31B taught me why raw accuracy is a trap. On paper it ties the leader on coding. In practice it runs at 12 tokens per second, six times slower than its smaller sibling, for three extra points. The culprit is architecture: dense models ran five to seven times slower than mixture-of-experts models of similar size on this machine, which completely reverses the ranking you’d guess from a parameter count.
The agent track held a counter-intuitive result of its own: tools raised scores rather than lowering them. The two Qwen3.6 models gained 10 and 8 points going from one-shot to agent, because writing code straight to a file sidesteps the repetition loops that eat their one-shot answers. Agents usually lose points to orchestration overhead; here I watched the opposite. Honesty about my own rule, though: the agent column is a single pass, not three, so treat gaps under about three points there as noise. A ten-point gain clears that bar easily. The one-point differences in the table do not.
One open thread: a much smaller model, Qwen3VL-8B-Uncensored, has so far only run the reasoning track. There it tied the 31B gemma overall and beat every model except gpt-oss at tracking sequential state, at roughly a quarter of the parameters. If it holds up on the coding tracks, the value calculation changes again. That’s the point of a repeatable harness: finding out costs one afternoon.
Before this project I would have told you that running a model at temperature zero (the “deterministic” setting) gives you the same answer every time. It does not, at least not on llama.cpp. Batching, expert routing in MoE models, and quantised caches make the arithmetic depend on server state. I sent five identical requests for one problem and got answers of wildly different lengths scoring 0, 0, 11, 0 and 11 out of 11.
The consequences are brutal for anyone quoting single-run benchmark numbers. My first sweep ranked three Qwen models 63, 53 and 49: a clear ordering, a clear winner. Three passes later, all three converged to an identical median of 57. The original ranking was pure noise. Single-run error reached ±11 points on a 68-point scale, and only half the fleet is calm enough that one pass would have been safe: gpt-oss, gemma-4-31B and Qwen3-Coder vary by two points or less from pass to pass, while the rest swing by up to 11. You cannot know which half you are holding until you have run it three times.
So the method now has a rule: run everything three times and publish the median with the spread visible. And a corollary I learned from the reasoning track: item count beats repetition. The 102-item reasoning exam produced pass-to-pass spreads of 0 to 2 points, because lots of small binary questions average the noise out. If you want a stable benchmark, don’t write cleverer questions; write more of them.
Measuring local models fairly turned out to be full of tripwires, and each one, undetected, would have produced a confidently wrong conclusion.
Reasoning mode can destroy a model’s score. Under greedy decoding, several models fell into repetition loops while “thinking”. One spent its entire 16k token budget repeating a single verification line and returned an empty answer, scoring 0 on a problem it otherwise handles well. Disabling extended thinking fixed it and was 18 times faster. Any score quoted for these models has to say whether thinking was on.
A model can be capable and still fail your setup. Qwen3-Coder emits its tool calls in its own syntax that llama.cpp doesn’t parse. An off-the-shelf coding agent would watch it do nothing and score it zero. My harness recovers those calls, taking it from 0/68 to 57/68, but the gap between those numbers is an integration problem, not an ability problem. Both facts matter if you’re choosing a model to actually use.
Your benchmark can lie to you before any model does. A self-test pass over the exam caught a bank of trap questions in which every correct answer was “B”. Left in place, it would have crowned the model most willing to guess B as the best reasoner. If your exam is built rather than borrowed, the exam needs a grader too.
Benchmarks rot. One of the six reasoning categories is now dead: every model scores perfectly on abstention, and two more categories are nearly there, aced by all but one or two models. Saturated categories contribute free points that compress the real differences. A model even saturated the whole track: gpt-oss went 102/102, so the exam can now only rank everything beneath it. Strip the saturated categories and the real contest is over just two: state tracking and constraint satisfaction, 40 of the 102 points. Read that column of the table with this in mind. A living benchmark needs pruning and hard new items the same way a codebase needs refactoring.
Because the alternative is choosing your daily-driver model from launch-day marketing and other people’s vibes, measured on hardware you don’t own, at settings you’ll never run. A local harness flips that: same prompts, same graders, same machine, three passes, medians with spreads. When the next model drops, the evaluation is one command, and its score means something because everything before it was measured the same way.
The conclusions will be yours rather than universal. My speed numbers are my machine’s, and yours will differ. But that’s the feature. The question was never “what’s the best model?” It was “what’s the best model for this box, at this size, at this speed?” And that question, it turns out, you can answer in an afternoon.
The results are public: github.com/kcsinclair/agent-benchmark has the scoreboards, the spreads, and the methodology notes, and new models will land there as they are measured. The grading harness itself (five graded coding problems, a seeded reasoning exam, an agent track and a speed sweep) stays limited access, precisely because public graders become training data. If you want access, ask. And if you want the build-your-own walkthrough, say so in the comments and I’ll write it next.
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.