Everyone's Arguing About Models. The 30x Was the Harness.

A quick note before we start: last Monday's issue accidentally went out to paid subscribers only. That was on me, so here it is, free for everyone: Claude Can Record Your Skill. It Can't Record Your Judgment. Now, to this week.

Last week, Composio ran an AI model through three different setups on the same 28 tasks, and only one number really moved.

On quality, the three setups basically tied: the model finished 22 of the 28 tasks in one, 21 in another, 20 in the last. And the model itself never changed. It was the same one every run, Kimi K3, not a frontier flagship but an open model anyone can download and run.

What moved was the bill.

Depending on which setup was wrapped around that identical model, the same task cost up to 30 times more to run. Same model, same tasks, nearly the same results, and a 30x gap in the cost. All of it came from the actual setup.

Composio@composio

We ran Kimi K3 through 3 agent harnesses (Claude Code, Hermes, Kimi Code) on 28 identical tasks. All 3 harnesses completed the tasks at similar success rates, but the interesting story is token efficiency: the same task cost up to 30x more tokens depending on the harness. 🧵🧵

1:04 PM · Jul 29, 2026 · 673K Views

220 Replies · 174 Reposts · 2.89K Likes

In a recent article, Thariq Shihipar, an engineer at Anthropic, put it about as plainly as this conversation ever does: a harness is the program that wraps the model, deciding what it reads, when it acts, and how its output gets checked.

Thariq@trq212

https://t.co/R6exTuF7P8

8:26 PM · Jun 2, 2026 · 3.27M Views

272 Replies · 1.43K Reposts · 10.9K Likes

That’s the readable version. But most of it isn’t. The benchmark, a 110-paper survey, the threads stacking up all week—nearly all of it written by engineers, for engineers, in words that quietly lock out the people who’d benefit most.

In reality, a harness sound like a coding idea when, upon closer inspection, it’s really more of a knowledge work idea. And while you might not think it, you’ve probably built a rough one without knowing there was even a word for it.

Most people who feel like AI keeps handing them slop reach for a better model or a better prompt. Neither, however, is what the above benchmark pointed at. Remove the jargon out of "harness" and what's left is four things any knowledge worker can hand a model:

1. A role,

2. A sequence,

3. A definition of done; and

4. A file it reconciles against before it moves.

It's the reason the same model gives one person a usable draft and another slop. The model is identical. The “room” around it does the rest.

I found this out by getting it wrong.

A few weeks back, I was reading an early draft of a newsletter when something felt off. I asked my pipeline why the editor step that owns positioning hadn't weighed in. Turns out the model had made those calls itself, marked the step done, and moved on. Two days later, it skipped again. Twice in three days, a step reported as done that never happened.

To stop that, I asked Claude to write a file. A one-page ledger it reconciles at the top of every turn: which step it's on, what "done" means, whether the last step's deliverable actually exists. A step counts as finished only when the thing it was supposed to produce is there to point at. The model announcing "done" doesn't settle it.

Since adding that file, my editorial pipeline has stopped skipping steps. The model I’m using is no smarter than it was in July (I’m using Opus 4.8); it just can't move on now until it checks its work against something it can't talk past.

Think about the last time you got real work out of a model. There was probably a role you pasted in, a doc you made it read first, a "don't do this" you repeat because it burned you once. That's a harness. You just never called it that.

And a real one isn't something you download. Mine took a year, one miss at a time. Nobody starts with fifteen steps and a definition of done. You start with the one rule you add the day the model burns you.

Not everything needs one, of course. Ask a model to reword a sentence and a role and a ledger are just ceremony: nothing to reconcile. The scaffolding earns its keep on the work you do again and again, where "done" is easy to fake.

Composio’s benchmark was never about the number, or the model, or which one to switch to next. The difference between usable work and slop was sitting in the setup the whole time, and you’ve probably already built most of yours without meaning to.

So, the next time an AI model hands you something unusable, don’t go shopping for a better one. Find the one piece of your harness that’s letting you down, and fix that instead.