I Don’t Use One AI Model for Everything. I Built an Agent Harness Instead.

One model may be best at planning. Another may be faster or cheaper for routine work. A local model may be the right choice for private context.

The problem is that switching models often means switching more than the model. The interface changes. Context gets stranded. Project instructions need translating. Tools behave differently.

I wanted the choice without losing the workflow.

So I standardized on a different layer: the agent harness.

In my Pi setup, that harness keeps the durable pieces in one place: context, skills, tools, state, safety gates, and evals. The model is a route through the system, chosen for the job.

That does not make models interchangeable. They still need different settings, tool schemas, and adapters. The point is to isolate those differences instead of letting them reshape the entire workflow.

My current harness has 15 specialized skills and six direct extensions outside the active model. The question I use to test it is simple: can I choose a different model without losing the system that makes it useful?

Community threads ask whether people are “switching AI models depending on the task” and “what happens to your context?” when they do. On Hacker News, one team built a model router after costs climbed because “we didn’t need Opus for everything”.

Choosing a model is not the hard part. Keeping the workflow stable enough that the choice stays cheap is.

Model independence cannot mean pretending every model behaves the same. It means keeping the stable workflow in one place, isolating model-specific differences, and testing a new model before promoting it.

That distinction matters. If your operating rules live in one vendor’s hidden memory, your tools depend on its proprietary interface, and your history exists only in its account, you do not have a model strategy. You have a tenancy.

The cost is not only another subscription. It is the time spent rebuilding permissions, context, tests, and team habits inside a new product.

Ask one question:

Can I use a different model for this workflow without rebuilding the system around it?

I use five checks:

1. Context: Are the durable instructions in files I can inspect and version?

2. Capabilities: Are recurring workflows encoded as portable skills or typed tools?

3. State: Can I retain and inspect work history outside the provider?

4. Adaptation: Are model-specific settings isolated from the rest of the workflow?

5. Evidence: Can I run the same representative tasks against a replacement model?

If the answer to all five is yes, choosing a different model is an engineering task. If two or three answers are no, the choice comes bundled with migration work.

My setup keeps reusable skills in versioned files, adjusts six sampling fields for one local model through an extension, and launches fresh-context agents for research, implementation, and review. Those controls live outside the active model.

The paid result is one written routing decision for one workflow: keep the current model, promote the candidate, or split the work between them.

You will leave with:

• A Harness Ownership Map that reveals where your context, tools, and state still depend on one provider. The first useful diagnosis takes about 15 minutes.

• A minimum harness setup with durable project rules and one reusable skill. The walkthrough uses Pi, but the map and test apply to other agent tools.

• A five-case Model Swap Test, complete seeded fixture, hidden answer key, two result receipts, and a promotion rule.

If two model routes already work in Pi, the full path should take 60 to 90 minutes. Otherwise, complete the map first and treat provider setup as a separate prerequisite.