Agent harness design

1.1 Where the rest of this lives

Six pages split the agent stack between them, based on which layer each covers and which domain that layer serves.

Reasoning and proof models falls outside the grid: it is about using LLMs for mathematics, and in particular the unusually clean verification that maths permits.

2.1 Two kinds of executor

Executors divide by whether running one leaves anything behind. A side-effecting executor does work — it edits a file, applies a migration, pushes a commit — and the artefact it leaves behind is the point of running it. A side-effect-free one only checks, and nothing outside the check changes. Only the second kind can be run over and over: fifty attempts at a proof cost fifty times the tokens and nothing else, where fifty attempts at a database migration is a different sort of afternoon. That asymmetry is most of why maths loops sample so freely and why nobody fans out an agent with commit access.

The property usually tracks the domain but is not the same thing as it. A coding executor is side-effecting by default, and the same edits made in a throwaway git worktree or a container we discard are not, which is what that tooling exists for. Maths gets the property for free; coding has to build it.

A second and independent question is what the executor hands back. What comes back is either raw material for the next attempt or a judgement on the attempt just made — a verdict — and the same tool emits both at different moments. Requena et al. (2026)’s prover works in Lean, whose compiler machine-checks every step and will also compile a proof with declared gaps, reporting what remains to be shown for each. A compile with no gaps is a verdict: theorem proven, stop. A compile with gaps is a to-do list for the next attempt.

3.1 Wiring

Wiring is the “shape” of the call graph — how many steps, in what order, feeding what. A step here is one node in that graph, which is not the same unit as a turn: fifty steps can run at once, where turns only ever come one after another. There are four things a step can do: chain onto the last one (sequential), fan out into copies of itself (parallel), route to one of several branches, or recurse into a fresh sub-problem. Only the first two are pure wiring, and they are what the rest of this section is about. The other two are where wiring reaches into the neighbouring axes and are treated there — a route is a dispatch decision given structural form, and a recursion is a sub-agent — a child loop with its own history — which is a context move.

The two pure forms: parallel samples many attempts at once and aggregates them; sequential runs one attempt at a time and feeds each result into the next. Parallel pays when attempts fail independently and a verdict can cheaply pick out the winners; sequential pays when each failure teaches the next attempt something. The two also spend differently: parallel buys attempts with tokens and gets them back in wall-clock, while sequential buys them with wall-clock and a context window that grows every turn. Fan-out width is a budget decision as much as an accuracy one. It can be a resumability decision too, once a runtime is checkpointing: Claude Code’s workflow runtime replays cached results in the order agents started, stopping at the first one that did not finish, so everything launched after it re-runs even if it completed. Stopping mid-fan-out therefore costs less the finer the fan-out is, and “many small agents or one long one” turns out to be a question about interrupted runs as well as about accuracy. In practice we increasingly use hybrids and compositions of these hybrids (Zhang et al. 2025), which come with vendor names — prompt chaining, routing, evaluator–optimizer and the rest.

The parallel half of this is only available where the executor is side-effect-free. Fifty proof attempts are fifty times the tokens; fifty attempts at the same repository are fifty conflicting sets of edits. So the domain sets the default, and a designer who wants the parallel half has to pay for the isolation that makes it safe — which is half the design space, gated behind a decision most people never notice making.

Parallel execution entails an aggregation step — majority vote, best-of-\(n\) against a verifier, filtering out the false answers, LLM-based consensus.

Aggregation is not always available, and whether it is depends on what the branches return. Comparable outputs — a number, a label, a classification, a boxed answer — have a mode, and the mode is a signal. Arguments do not. \(k\) candidate proofs, \(k\) design proposals, \(k\) code-review write-ups are \(k\) different objects, and a set of arguments has no modal element to take. Where the branches return arguments, parallelism still buys candidates, but the aggregation step has to become a judge or a checker rather than a tally — a different and much more expensive component, and one that can be wrong in ways a tally cannot. The fan-out is the cheap part; the thing on the end of it is not, and what that thing can be is the next axis. The maths case is where this bites hardest, but it is not a maths-specific problem.

The two forms compose rather than compete. Requena et al. (2026) test single-shot parallel sampling against sequential refinement, and find that adding refinement to a parallel fan-out loop improves sample efficiency — more theorems proven per token spent.

The other thing wiring fixes is how much of the control flow is guaranteed. A hand-written script that branches and retries does the same thing twice; an agent asked to follow a SKILL.md mostly does. That determinism trades against the flexibility that made an agent worth using in the first place, and it is what an agent emitting a script instead of taking turns is buying.

3.2 Verification

The thing being judged is a candidate — whatever the model has offered as an answer or as progress toward one, a patch, a proof, a plan. Wherever a step judges a candidate, two independent properties of that step are in play: the grade of the judgement — how far it can be trusted — and its unit — whether it is about a whole candidate, one claim inside it, one sub-goal, or one step of the argument.

Grade first, because there is no sense making multiple attempts at something if we cannot choose which attempts are better. It is two questions rather than one, and they have different consequences. Is the verifier complete — does it accept everything that is actually correct? And is it incorruptible — can a wrong candidate talk it round?

Incompleteness costs recall, and nothing else. An incomplete verifier throws away some correct candidates — a computer algebra system (CAS) refuses a pair it cannot simplify — but it cannot be selected against: no amount of sampling teaches a candidate how to be wrong in a way a CAS likes. A corruptible verifier can be. Sample far enough against a judge and we begin selecting for candidates that fool the judge rather than candidates that are right. So incompleteness sets how much of the budget gets wasted, and incorruptibility sets how large the budget can usefully be — which is to say, the second question is the one that licenses sampling wide.

Four combinations, four names:

Against an exact verifier, success keeps climbing with the number of attempts (Pass@k in the benchmark jargon: the chance that at least one of \(k\) samples is the real deal), and prover loops sample in the thousands. Against a soft judge, if we are concerned about reward hacking at inference time, it becomes reasonable to allocate effort into improving the judge instead of widening the search. Nomos is the worked example of taking that option: a judge trained alongside the proposer, and a pairwise tournament where the vote used to be.

The soft grade also has a ceiling no judge clears: wherever aggregating means deciding that two outputs say the same thing, that decision is a semantic equality test, and semantic equality is not in general decidable.We can usefully combine grades, it turns out: Requena et al. (2026) back the compiler with deterministic checks for the known dodges (which apparently is a thing) and then use an LLM reviewer for the residue — statement-weakening and metaprogramming tricks that compile without proving the theorem. A soft verdict layered over an exact one buys something neither has alone.

Now the unit. A verdict on a whole candidate tells the loop only whether to keep it; a verdict on one claim inside the candidate tells the loop which part to fix, which is what gives a sequential wiring something specific to iterate on. The two properties interact in one direction: a fine unit is only affordable when the grade is cheap and incorruptible, because a fine unit means calling the verifier hundreds of times per candidate, on fragments, and a judge asked to grade a single step answers no better than it grades the whole thing. That is why the finest-grained systems all sit behind a compiler and the coarse ones are where soft verdicts live. Mathematics has climbed further down this axis than anywhere else — the rungs, and which systems sit on them.

3.3 Context

What each call sees, i.e. context engineering. Every call to a model endpoint is a fresh call, so the “history” the model brings to the current conversation is a choice, remade each turn. Traditionally that is the system prompt and any skills, some tool definitions, retrieved material, the history so far, and whatever notes the loop has kept. We have considerable scope to vary it.

This breaks in all kinds of interesting ways. Too little context and the model wastes time re-deriving what it worked out an hour ago. Too much context, aside from being expensive, can fail in various complicated ways: we can just run out of context window, attention to the middle of a long context degrades (the lost-in-the-middle effect), and we increase the risk of propagating bad ideas ad infinitum.

The responses available are three verbs, and everything this page later calls memory is some composition of the five moves under them.

Fetch — leave it on disk, pull it in when wanted.

RAG

pulls in knowledge the model never had in the weights — Mathlib search, repo search, web search.

Externalization

does the same for knowledge the loop produced itself: write the state to a file, keep a pointer in the window.

Shed — cross a window boundary by throwing something away.

Compaction

summarizes the conversation so far, discards the originals, and starts a fresh window with the summary at the front.

Rebuild

discards the window outright and seeds a fresh one from externalized files, so nothing is ever a summary of a summary. Xiaomi’s term, and the move that session memory turns on.

Delegate — spend a different window.

Sub-agents

take a self-contained sub-task to an isolated child — its own conversation history, its own tool budget — and return a summary, so the dirty work (grepping code, searching the web, writing throwaway spikes) happens in a context we then throw away and the parent’s stays lean enough to keep planning. Claude Code supports these natively as agent teams; the pi harness leaves them out deliberately, for the reason below.

That is the whole vocabulary. What changes as the timescale stretches is not the moves but which of them we compose and who is allowed to run them, which is the long-horizon problem.

Requena et al. (2026) ablate this axis against no memory at all, a rolling history of recent attempts, and a self-managed note the model writes for itself. The note wins on both theorems proven and cost.

3.4 Dispatch

We could use a single frontier model for every step, but typically need not. Specialized models cover specialized capabilities, and cheap ones cover the easy steps to keep the cost down. In the hand-built loops the answer is hard-coded. We give the agent a solve() oracle and let it decide when to call it, or wire the specialist into the script ourselves, or pick a fast model for the sub-agents and a slow one for the planner. Of the four axes, dispatch is the cheapest to change — usually the only free one — which is why the interesting arguments are about the other three. We are also free to learn to make this decision rather than hard-code it.

3.5 How the four interact

Wiring is the primary choice, and verification the primary constraint. Wiring is what a designer actually decides, and the other three answer questions it raises: every wiring needs a verdict of some sort, since parallel needs one to select among attempts and sequential needs one to inform the next attempt and to decide when to stop. But the grade of verdict available bounds both shapes, and the grade is mostly handed to us by the domain rather than chosen. So where the domain fixes what a verdict can be — as mathematics does, emphatically — the dependency runs the other way: the verdict we can get decides the shapes we can afford, and that was settled before we arrived.

Sequential wiring additionally drags in context, because attempt \(n+1\) that remembers nothing of attempts \(1..n\) will make their mistakes again, so the memory design is what stops a sequential loop going in circles. Dispatch is the one that really is orthogonal — whatever the shape, we still choose who fills each slot.

That is why verification and memory keep turning up together in ablations despite looking unrelated: one is what any wiring needs in order to know whether it is getting anywhere, and the other is what the sequential kind needs in order to get anywhere at all.

3.6 Systems as coordinates

Factored this way, harnesses built for different jobs are points in one space.

The coding agents crowd into one corner while the maths agents spread across the space. The always-on assistants answer verification with nothing at all, which is why they suit open-ended errands and are dangerous left alone. The last row cheats: Ornith is not a point in the space but a search over it, below.

What these four choices become once the answer is a piece of mathematics is in the reasoning notes; what I built out of them is a six-step path.

4.1 Memory within a session

One long conversation about one cohesive project — the app I am building, my tax return — outlives its window well before it outlives its usefulness. Session memory is whatever policy we pick over the five moves to carry it across that boundary.

The default policy is compaction and nothing else. We might imagine this could even be beneficial, because the summary might skip over red herrings and false turns in the original, no? In practice many things can go wrong. Compaction degrades on long tasks, because each summary is a lossy compression of the previous summary and the losses compound the same way the per-turn errors do.

Two policies that do better.

Let the human pick what survives. Still compaction, but the cut is chosen rather than automatic. pi’s /tree jumps back to an earlier point in the history and collapses everything after into a précis, making what counts as “currently” relevant context an overt user choice.

Move memory out of the main loop. Externalization and a sub-agent and a rebuild, with compaction dropped entirely. The main agent keeps almost no notes of its own; a separate writer subagent, dispatched by the runtime rather than invoked by the model, reads the conversation and commits structured checkpoints to disk. It fires early — well below the limit — so the extraction happens while there is still room to think, and before lost-in-the-middle erodes the thing being extracted. When the window does fill, the runtime rebuilds from those files instead of summarizing what is in it, so from the model’s gaslit point of view the conversation never breaks.

MiMo Code is the fullest worked example I know of, and Xiaomi document it in enough detail to copy. The writer fires at roughly 20%, 45% and 70% of the context budget, each pass an incremental update to the last rather than a fresh summary of a summary, which is what breaks the compounding-loss chain. Its output sits on four lifecycles: checkpoint.md for this session, MEMORY.md for this project, a global file for this user, and an unindexed SQLite trace of every message and tool call underneath. That bottom layer is the observability hedge, since everything above it is a summary written by the thing we would want to audit. Each structured file has exactly one writer, enforced in code rather than by prompt; the main agent’s one channel is a free-form scratchpad that the writer drains and clears at each checkpoint.

The general form is older than the coding agents and has a better name: Letta’s sleep-time compute, in which the primary agent is deliberately not given the tools to edit its own memory and a background agent gets them instead. OpenClaw cites the same work for its consolidation pass. Qwen Code looks like this and is not, which cost me some confusion: it does run a background extraction agent over the transcript, but it fires once per turn rather than at budget fractions, it harvests durable facts about the user rather than what the agent is in the middle of, and when the window actually fills it compacts like everyone else.

4.2 Multi-turn reliability

A long-horizon run is a chain of individual turns — read context, decide, call a tool, repeat. Each turn accumulates some additional chance of a wrong call: editing the wrong file, passing a test by changing the test… Those per-turn odds compound, so even a low rate dominates a long enough run — at an i.i.d. error rate of 2% per turn, a 200-turn task finishes clean only about 2% of the time (\(0.98^{200}\approx0.02\)). A short interactive session is easy to correct — we, the humans, spot the bad turn and fix it. But the economic benefit that agents are being sold to us on is doing unattended multi-hour runs. In that case, we really want to juice that error rate down low to top out the famous reliable task-length metric.

A compounding error rate has exactly three attack surfaces. We can make each turn less likely to go wrong, make the chain shorter so there are fewer turns available to go wrong, or catch the wrongness at the end and go round again.

• Buy down the per-turn odds, and spend to do it. Sample several candidate plans in parallel and use a low-temperature judge to pick one, aimed at the planning turns where a wrong decision costs the most downstream. A marginal gain at a large cost, which might still be the right trade on a high-stakes task.
• Shorten the chain by emitting code instead of taking turns. The agent writes a script — spawn an agent, run these in parallel, pipeline those — and a sandbox runs it deterministically. This is the wiring choice again, with branch and retry logic guaranteed by code rather than by a model remembering to follow a SKILL.md, which means many fewer per-turn decisions are available to go wrong. That code-beats-prompt principle for predictable control flow is originally Anthropic’s, and Anthropic now ships the worked version of it too.
• Catch the failures at the end with a stopping-condition verifier. The user states a done-condition in natural language, and each time the agent tries to stop, an independent model call checks the whole history against it; if the work is not finished, it whinges about the shortcomings and tries again. This is the disciplined form of the Ralph Wiggum loop — the brute-force pattern of relaunching an agent on the same prompt until the job is done.

Two systems have converged on the second one, down to the name. Xiaomi calls it Dynamic Workflow; Claude Code calls it dynamic workflows, also JavaScript, also agent() and pipeline(), written by the model and executed by a runtime outside the conversation. The docs frame the distinction better than I had: what separates a workflow from a subagent is who holds the plan — a subagent leaves Claude deciding turn by turn with every intermediate result landing in the window, where a script holds the loop, the branching and the intermediates itself, so only the final answer comes back. That makes it a context move as much as a wiring one: emitting code is externalization applied to control flow. It is also where the quality patterns get somewhere to live — adversarial cross-review of findings before they are reported, or drafting a plan from several angles and weighing them — because a script can express “then check this” in a way a prompt can only ask for.

Note what the third one needs — an independent judgement of whether the work is actually done, stated in natural language and evaluated by a model. That is a soft verdict guarding a long and expensive run, with all the foolability that implies, and the longer the run the more opportunity the agent has had to talk itself into being finished.

4.3 Learning across sessions

OK, now ultra-long horizons. Not just tasks, but whole projects. Or whole careers, but let’s not get ahead of ourselves.

If we come back to the same project regularly and each time the agent has forgotten everything, it feels like we’re leaving performance on the table, re-deriving the same constraints and repeating the same mistakes. A new session starts empty, so everything it knows it reads off disk, and the interesting question is how anything got written there.

Not by being composed at project scale — by surviving. MiMo’s writer moves an observation from checkpoint.md up into MEMORY.md once it has held steady across several session checkpoints, so the project file is a filter over the session files rather than a separate act of authorship.

What is new at this timescale is rot. A session checkpoint is overwritten every few hours and never lives long enough to go stale, where a project memory accretes for months, its file references decay, and no one in the session loop is looking after it. So we get background agents grinding over the agent’s own history, and they want two clocks rather than one. The faster one tends the facts — MiMo calls it Dream: weekly, it reads past sessions and the memory files, merges and deduplicates, checks that file references still resolve, and compresses the result back down before it sprawls. The slower one harvests process rather than facts — Distill: monthly, it hunts for recurring work patterns and solidifies them into reusable skills, CLI commands, and standing procedure documents.

Only the first of those is still memory. A skill changes what the agent does rather than what it knows, which is why this section is learning and the one above is not — and it is the same move the next one makes on the harness itself.

The dreaming half is sleep-time compute again with the period stretched from minutes to weeks, and enough people have arrived at it independently that they have converged on the name: MiMo, Qwen Code and Letta all ship a dream pass, with Qwen’s gated on a day having passed and five new sessions having accumulated. Hermes points the same machinery at the user rather than the project, accreting auto-generated skills, FTS5-searchable session history, and a persistent model of our preferences via Honcho’s dialectic user-modelling, on the pitch that the longer it runs the more it knows about us. OpenClaw keeps persistent memory in the same spirit, and cites Letta for it. Either way the library is now writing itself, which is a standing risk, and a worse one here than within a session: a bad session checkpoint dies with the session, where a bad skill is loaded into every session after it.