I built a benchmark to test my own harness against a bare setup and Garry Tan's gstack. It costs four and a half times the bare one — and that's the best money I spend, because it lets me run a fleet of agents and walk away.
I built a benchmark for agent harnesses, pointed it at my own, and it came out the single most expensive thing in the field — four and a half times the cost of a bare setup. I would pay more. Because on every hard task, my harness quietly spun up a security reviewer, a code reviewer, and a documentation writer, and shipped the docs the cheap runs never wrote. The tokens are not the expensive part. My attention is — and that is exactly what it buys back.
Here is the claim I want to leave you with: you cannot judge an agent harness by the code it emits. The code is the one thing every serious harness already gets right. What separates them is the work that happens around the code — the security pass you didn't run, the review you didn't ask for, the docs you forgot, the convention it held while you were in another window. That work is the whole job, and it is invisible in a diff. So "which harness is best" is not a number you rank. It is a question of how much of that job you want taken off your hands, and what you'll pay to stop thinking about it.
This matters now because the bottleneck moved. You are not typing the code anymore; you are holding the whole surface — is this secure, is it tested, is it documented, did it drift from what we agreed — in your own head. That vigilance is the real cost of AI-assisted work.
And it's worth stating the actual goal plainly, before the numbers start: I am not trying to squeeze a better answer out of one agent. I am trying to run a fleet of them — a dozen workers going at once, each on its own task, while I am somewhere else. That only works if every worker can be left alone, and a worker you have to babysit does not scale to twelve. You become the bottleneck the moment the second one needs you. So the whole game is building workers that police themselves — review their own security, check their own quality, write their own docs, hold the conventions you set three sessions ago — so you can launch them, step away, and come back to judgment calls instead of cleanup. Everything the benchmark measures is really a proxy for one question: can I leave this thing running unattended?
The tool is called agentic-benchmark, and the thing under test is not a model. It is a doctrine — a CLAUDE.md plus whatever skill bundle ships with it. An executor running on Sonnet writes against a fresh git workspace; a fixed Opus judge, with no tools of its own, scores the result. There are five axes — Correctness, Adherence, Craft, Efficiency, Reliability — and they are never summed into one score. Everything that can be checked deterministically is checked by the harness itself: tests pass or they don't, files land inside the expected surface or they don't, duplication and debug logs get counted by machine. The judge only scores the qualitative residual, and even then it scores head-to-head instead of on an absolute scale, because a comparison is far less noisy than a lonely number.
Three variants went in. My own, agentic-os. A naked baseline — a competent but bare CLAUDE.md. And gstack, Garry Tan's harness. Six real backend tasks, three runs each, so I could see variance instead of pretending one run is the truth. One caveat worth stating up front: every run is a single agent on a single task. The fleet is what this per-task autonomy enables, not something the benchmark clocked directly — so read the numbers below as evidence that each worker can be left alone, which is the thing the fleet is built on.
Start with the gate. On correctness, all three tied — every variant passed three of three on the tasks with executable checks. If you scored these harnesses on "does the code work," you would call it a wash and go home. The whole interesting story starts exactly where the diff ends.
The benchmark has a campaign mode built for that: it establishes a convention, forces a context reset, and then checks whether the convention survived the reset. Across that chain, agentic-os was the only variant that held it — once by abstraction, meaning it reused the underlying rule rather than parroting the literal convention back, which is the strongest memory signal the benchmark tracks. naked and gstack fell into the known trap on every single run.
I am going to be careful here, because this is the axis I most want to be true and therefore the one I trust myself least on. agentic-os did not hold every time. Of its three campaign runs, one held cleanly and two mostly walked into the trap. And on a separate memory probe, all three harnesses — mine included — hit the same wall. So the honest claim is not "agentic-os remembers." It is: agentic-os is the only one of the three that can hold a convention across a reset at all, and when its memory fires, it holds by abstraction. That is a real edge and a narrow one, and it is what lets a worker keep your conventions without you in the room to restate them.
Here is the part the score sheet files under "not measured," and it is the part I actually pay for. On the three hardest tasks, agentic-os did not just write code. It dispatched an engineer, then a security reviewer, then a code reviewer, then a documentation writer — up to six sub-agents on a single task — and produced documentation files the bare runs never bothered with. naked and gstack spawned zero sub-agents and shipped no docs on those tasks. Same problem, same model underneath. One of them handed back a reviewed, documented result; the others handed back a diff.
That is why the craft comparison went the way it did. The benchmark scores craft head-to-head and weights a caught soundness or security bug three times heavier than a naming nit — and agentic-os won that comparison decisively, 76 to gstack's 68 to naked's 32. But the score is downstream of the behavior. The reason it writes safer, better-documented code is that it ran a security review and a code review you never had to ask for, and wrote the docs you would have written later or never. You can vibe your way through a task and still get back something that was checked. That is the meta-game, and no single axis captures it because it is smeared across all of them — and it is exactly the property that lets you point a worker at a task and not read over its shoulder.
That behavior is what makes the harness hands-off, and hands-off is the whole point. A harness that reviews its own work and holds your conventions across a reset is a harness you don't have to stand over. And a harness you don't have to stand over is one you can run ten of at once — which is the only way the parallel math ever closes. The win isn't a faster single agent. It's an agent that doesn't call you back. The behavioral trace supports it: agentic-os ran fewer top-level turns than either competitor — 244 against roughly 350 — because it fans work out to those sub-agents instead of grinding through one long loop you have to steer. Fewer top-level turns is a fair proxy for "you touch it less." I'll flag that as an inference from the turn counts and the sub-agent usage, not a figure the benchmark measured directly, because the strongest objection to it deserves a straight answer.
That objection comes from gstack. Garry Tan built it as a twenty-three-specialist harness he calls a virtual engineering team, and its stated ethos argues the opposite of hands-off. Its "User Sovereignty" principle holds that experienced engineers should interrupt the model more, not less — that when you remove yourself from the loop, you stop knowing what your agents are doing. It is a good argument, and it is right about one kind of interruption. There are two. There is interrupting for judgment — taste, direction, the call the model cannot make — and there is interrupting for babysitting: re-explaining a convention it dropped, re-checking work nobody reviewed. Tan's ethos is about the first, and you should never automate that away. agentic-os is aimed squarely at the second. It kills the babysitting so you have attention left for the judgment — and so you can spend that judgment across ten streams instead of burning it re-steering one. Those two kinds of interruption are not in competition; confusing them is how you end up defending the wrong one.
So here is the bill, and I am not going to soften it. agentic-os spent $34.86 across the six tasks. The bare baseline spent $7.94. gstack spent $13.92. My harness is the most expensive option by a wide margin, running about four times the output tokens of the bare run. And the cause is not waste I can trim — it is the security reviewer, the code reviewer, the documentation writer, and the sub-agent fan-out that produced everything above. The care costs money, and every bit of it lands on the invoice.
But look at what the invoice is for. Twenty-seven dollars of extra tokens against the security pass you'd run by hand, the review you'd do yourself, the docs you'd write later or never — times every worker in the fleet, times every hour you'd otherwise spend supervising them. The tokens are the cheapest thing in that trade. What you're buying is the right to not hold the whole surface in your head — to start a task, walk away, and get back something reviewed and documented instead of something you now have to audit. If your bottleneck is your own attention across more parallel work than you can personally watch, four and a half times the tokens to reclaim it is not expensive. It's the bargain of the setup.
This is also where gstack earns its place. Tan's harness lands most of the craft — 68 to my 76 — at roughly forty percent of the cost, without the same fan-out. For work where you're staying in the loop anyway, that's the sharper buy. For work you want to hand off completely and trust came back checked, it isn't. The benchmark's job is to show you that fork honestly, not to pretend either side of it is free.
None of the memory behavior was luck, and it is worth saying how it got built, because it is the spine of the last three releases. agentic-os keeps a persistent MEMORY.md — the facts worth carrying between sessions — and for a long time both halves of that system leaned on the model choosing to do the right thing, which is exactly the thing a model stops doing as a session runs long, and the thing you cannot personally enforce across a dozen sessions at once.
v2.6.0 was cleanup with a lesson attached. I flattened CLAUDE.md into a generated file with the rules embedded inline instead of imported, and restored a block of dispatch doctrine an earlier cut had deleted — the deletion had correlated with the agents quietly forgetting to fan out at all, which is to say forgetting to run the very reviewers this whole post is about.
v3.0.0 made the write deterministic. An end-of-session extraction pass plus a Stop hook now nudge the still-live agent to record what it learned, against a sharper rule for what even earns a memory: save it only if it is decision-relevant and could not be re-derived from the repo. It is a breaking release — it also tore out a Cursor integration that had been a failed experiment — but the write path stopped being something the model had to remember to do.
v3.0.1 closed the other half. A SessionStart hook now injects the MEMORY.md index at the top of every session, framed as reference data, so recorded facts actually re-enter context instead of decaying in a file the agent never re-reads. Write became deterministic in 3.0. Read became deterministic in 3.0.1. That is the arc — and the benchmark's campaign chain is the first outside test of whether it worked. The answer so far: it works, narrowly, not yet every time. Which is about what you'd expect from a mechanism this young, and precisely the kind of thing a benchmark exists to keep honest.
I built the benchmark to refuse a single score because the number would have lied in both directions. It would have let me claim a win I didn't earn on cost, and it would have hidden the win I did earn everywhere the cost bought something. What the axes can measure, they measure honestly — and my harness is genuinely the priciest one on the board. What they can't measure is the reason I pay it: I have not thought about whether an agent's output was security-reviewed, quality-checked, or documented in a long time, because the harness thinks about it for me — on every worker, at the same time.
So pick the harness that matches the axis you're actually paying for. If your bottleneck is throughput on well-scoped work you're supervising anyway, buy the leaner one. If it's your attention — spread across a dozen parallel streams you cannot personally babysit — then the harness that reviews itself, documents itself, and remembers your conventions while you're off doing something else is worth every token, and the token bill is the least interesting thing about it.
The benchmark is open, and it will bill your harness the way it billed mine. Clone agentic-benchmark and score your own CLAUDE.md — it runs on the model access you already have. Star agentic-os if you want the harness that does the work you keep forgetting to, and open a pull request with your own variant if you think it can take the field on an axis that matters. And subscribe below, where I write up what this system teaches me, usually by going wrong first.
— Glenn Eggleton builds agentic engineering systems and writes about what survives contact with production.
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.