Harness, intelligence and compositional generalization

🔗 Learn more about me, my work and how to stay in touch: maeste.it: personal bio, projects and social links.

A week where I pick up the harness thread from where I left it, but I go down a level. Last week I showed the numbers proving that the harness counts more than the model. This time I go look at what’s inside it and why it works, crossing two points of view that complement each other. Addy Osmani gives me the vocabulary: loop, harness, factory, and comprehension debt as the risk when you lose control of the outer loop. Alex Zhang gives me the formal proof: compositional generalization lives in the harness, not in the weights, and an RLM trained on short tasks generalizes to tasks 8 to 32 times longer. One tells me where the system breaks when you automate without verifying, the other what you gain when the harness is well designed. In the links you’ll find multi-model routing, Kimi K3’s open MoEs, Gemini 3.6 Flash, Claude Opus 5 and Qwen-Image-3. Happy reading.

Podcast:

• On Saturday “Which skills we actually use for our agents” came out (#64 Risorse Artificiali): the skills we use with Claude Code, how we choose them, and why they aren’t written by hand but distilled from working sessions. Listen

Two voices, this week, saying the same thing I’ve been saying for a while, and saying it better than I was. Addy Osmani says it with the vocabulary of a builder, Alex Zhang proves it with experiments. The common point is my old fixation: intelligence doesn’t live only in the model’s weights, it lives more and more in the harness, the surrounding software that governs it.

Osmani starts from three layers worth keeping in mind. The loop is a single agent that does one thing and repeats it. The harness is the walls around the loop: the sandbox, the tools, the memory, the verification gates. The factory is many loops fed by a queue and drained into production through a review gate. It’s a clean hierarchy, and I like it because it gives a name to what we all do every day without calling it that.

From here Osmani derives what I think is the most important rule of all, back pressure: you can only delegate to a loop as much autonomy as you can verify cheaply and reliably. Generation is unlimited, verification is the bottleneck. It’s not a volume problem, it’s a problem of a surplus of bad PRs. It’s the same point I was making last week with Lilian Weng’s numbers, and before that quoting Simone Basso: the model is commodity, the harness is the asset.

Here Zhang comes in, and shifts gear. Osmani speaks as an engineer who has seen the factories; Zhang speaks as a researcher who wants to explain why the harness works. His thesis is strong: compositional generalization, the ability to solve new problems by composing familiar ones, should live in the harness, not in the neural network. A good harness takes a complex state and reduces it to small observations, each of which the model handles locally in-distribution, that is, within its own territory of competence. An unknown problem becomes an already-seen problem.

The concrete mechanism Zhang calls Recursive Language Model, RLM, and it rests on two legs. Context offloading: the specific context is passed as a symbolic variable and the root model doesn’t see it directly. And programmatic sub-agent calls: sub-agents are treated as REPL functions, whose output ends up in variables that the root doesn’t need to read. The result, every single call to the model stays clean, without the context rot that kills current harnesses like Claude Code or Codex.

Zhang’s numbers are what convinced me completely. He trains an RLM on short tasks and evaluates it on tasks 8 to 32 times longer, and it generalizes. He trains it on one domain and tests it on a completely different domain that shares only the latent structure, and it generalizes. The base Transformer, on the same tasks, stays flat even as the training reward grows. With Qwen3-30B-A3B and the RLM harness it comes close to or beats GPT-5.5. It’s the formal proof of what Osmani intuits from practice: harness design shifts performance more than we should expect.

Now I put the two voices side by side and I see the same lever from two sides. Osmani tells me where the system breaks when you automate without verifying; Zhang tells me what you gain when the harness is well designed, and gives me the proof that generalization is bought there, not in the weights. They’re two sides of the same coin.

And the point where they touch is precisely the outer loop. Osmani insists: the engineer must hold on to the outer loop, decide whether the approach is right, verify soundness, approve changes, carry the consequences. The inner loop, investigate, fix, test, you delegate. This is where comprehension debt plays out, the debt of comprehension, and I want to read it the way that convinces me most. It’s not, or not only, code that nobody reads. It’s the loss of control over the comprehension of the system, and above all over how the outer loop is built, made of triggers, verifiers and guardrails. When you stop understanding why the agent starts, what stops it, which boundaries govern it, the debt accumulates even if the tests stay green. And it’s exactly the governance rule I defend: the evaluator and permission control must stay outside the loop that evolves the harness, otherwise you break the abstraction boundaries and open the door to reward hacking. Short loops, 3-10 steps, verify well; beyond 20 the agent loses the thread. You earn autonomy with human judgment upstream, on design and architecture, you don’t just delegate it.

This paper speaks my language: when does it make sense to route between multiple models instead of calling the biggest one? You need real behavioral diversity and a stable policy, and few are enough: fewer than 10 agents capture almost all the available diversity, 4 in one benchmark. You don’t need a zoo, a curated and differentiated group is enough. Composing, yes, but the most accurate routers are also the most fragile.

I return to Kimi K3 for the trend of open MoEs. Total parameters have grown twenty times since Mixtral, active ones are almost flat, between 17 and 49 billion for over two years. Sparsity is paid in storage, which is cheap, not in compute and bandwidth, which are scarce. And there’s a half political admission: when FLOPs are rationed by export controls, you scale on the axis they don’t touch. More ammunition for the open insurance policy.

Google pushes on Flash and for me the theme is one, efficiency. Gemini 3.6 Flash outputs 17% fewer tokens, 65% fewer on DeepSWE, and Flash-Lite costs very little. In multi-agent systems latency and cost add up at every call: every token saved is one more composition I can afford. It’s not enough to have different models, they have to be cheap enough to combine.

Opus 5 closes the gap with Fable 5 at half the cost, with self-verification, lower variance and two features that speak to the harness: tool changes mid-conversation and fallback between models. It remains the usual black box, which limits how much I can build on top of it. And the connection to the deep dive is direct: a model that generates more code shifts pressure onto verification. The more capable it is, the more comprehension debt I risk if I don’t hold the outer loop.

I step out of coding for a moment. Qwen-Image-3 strikes me for the target, not for the quality: complex layouts, legible 10-pixel text, LaTeX formulas, infographic grids in a single pass, UI. When a component becomes this composable, it stops being a toy and becomes a piece of pipeline. It’s in composable components that multi-model finds its sense: the pieces specialize, and whoever knows how to assemble them wins.