RSS Amplifier

Artificial Impact · Aug 11, 2026

Designing an AI system vs integrating a model: why the distinction matters

Read inOriginalenesdefrptjaittr
0
Sign in to vote or save

Alexandre du Sordet · Artificial Impact

Photo: Tiago Ferreira / Unsplash — a Ferrari engine, mid-reveal. Magnificent, and still not a car.
Photo: Tiago Ferreira / Unsplash

Hello everyone,

A Ferrari engine is a marvel of engineering. Bolt it into a soapbox cart and you do not get a race car; you get a faster crash. In this article we are going to deep dive into the difference between integrating a model (bolting the engine into whatever you have) and designing an AI system (building the car around it), and why that distinction, more than model choice, decides whether your AI project survives. You know the scene: a dazzling demo built on one API call, validation, a pilot... and nine months later the same project is stuck! MIT’s 2025 enterprise study found roughly 95% of GenAI pilots produce no measurable P&L impact, and the failures concentrate not in the models but in everything around them. Let’s see what “around them” actually means.

Let’s be honest and generous first: the model API is a miracle. A general-purpose reasoning function, rented by the token, better every quarter. For a prototype, an internal assistant, or a first version, one well-prompted call is genuinely enough, and starting there is correct. Google’s Rules of ML has said it since before LLMs: don’t be afraid to launch without machine learning; start with the simplest thing. Anthropic’s own guidance on agents says the same: find the simplest solution possible, add complexity only when needed.

Remember the “just a GPT wrapper” insult of 2023? The prophecy came true, selectively. Thin wrappers died exactly as predicted: PDF-chat apps and summarizers evaporated when the platforms added file upload, memory and custom assistants as checkboxes. But two of the most mocked “wrappers” became giants: Cursor, the AI code editor, reached roughly $2B in annual revenue in about three years, with funding talks near a $50B valuation, and Perplexity, the AI-powered answer engine that took on Google search, raised at a ~$20B valuation. Cursor’s rise was then validated in the most dramatic way possible: in 2026, Elon Musk’s SpaceX acquired it in an all-stock deal valued at around $60 billion. Their secret is the whole point of this article: they were never wrappers. They built indexing, context assembly, routing across many models, their own evals, and obsessive product speed. The model was the engine. They built the car.

ℹ️ Small glossary

  • Orchestration = the logic that sequences model calls, tools, and checks into a workflow.

  • Retrieval (RAG) = fetching your own documents/data into the model’s context at question time.

  • Guardrails = automated checks on inputs and outputs (safety, format, policy, PII).

  • Evals = your scored test suite for AI quality; the unit tests of the AI world.

  • Routing = sending each request to the cheapest model that can handle it.

  • Data flywheel = the loop where corrections and feedback continuously improve your system.

  • Harness = the runtime scaffolding wrapped around a model: the loop that feeds it context, lets it use tools, checks its output, and decides whether to retry, continue, or ask a human.

So... is this new? Not remotely. In 2015, Google’s D. Sculley and colleagues published the most quietly influential diagram in machine learning: in real ML systems, “only a small fraction... is composed of the ML code”; the surrounding infrastructure is vast. LLM APIs shrank the small box even further. They did not shrink the diagram.

Berkeley researchers gave the modern version a name in 2024: compound AI systems. Their observation: “state-of-the-art AI results are increasingly obtained by compound systems with multiple components, not just monolithic models.” Even the headline benchmark numbers you attribute to “models” are system results. AlphaCode 2, DeepMind’s competitive-programming system, reached the 85th percentile of human competitors not through a smarter model but by generating up to a million candidate solutions and filtering them. And when Gemini famously topped MMLU (a standard exam-style benchmark covering 57 academic subjects) with 90.04%, the record came from an inference strategy orchestrating 32 calls and voting on the answers; prompted plainly, the same generation of GPT-4 sat at 86.4%.

Think of a virtuoso soloist versus an orchestra. The soloist is astonishing alone. But the record-setting performance is a conductor (orchestration), sheet music (prompts and policies), sections entering on cue (routing), rehearsals (offline evals), and the hall itself (retrieval and context). Gemini’s MMLU record was, quite literally, 32 performances conducted into one.

One word you will hear more and more is the harness (you will also see “agent harness” or “scaffolding”), and it is the piece most people miss. A raw model is like the engine: turn the key and it fires, but left alone it just sits there revving. The harness is the loop built around it that turns raw capability into useful work.

In plain terms, the harness is the code that hands the model the right context, lets it call tools (search the web, run a query, hit your API), reads back what happened, checks whether the result is any good, and then decides to retry, take the next step, or stop and ask a human. Same engine, wildly different behaviour depending on the harness around it. It is exactly why the same model can be a party trick in one product and a dependable worker in another: the model supplies the raw intelligence, the harness supplies the judgement about when to act, when to double-check, and when to give up. Anthropic’s own guide to building agents puts it bluntly: the systems that worked best were not the cleverest, they were the ones with the simplest harness that reliably did the job. When we talk about “designing a system” for the rest of this article, the harness is a big part of what we mean.

Four things your model provider cannot decide for you.

Errors in pipelines compound multiplicatively. At 95% per-step reliability, here is what your workflow’s end-to-end success looks like:

A 20-step agent that is “95% reliable” succeeds about a third of the time. Getting from 36% to shippable is not prompting; it is validation steps, checkpoints, rollback, and human gates. Architecture, in one word.

On January 27, 2025, DeepSeek’s cheap R1 release wiped a record ~$589B off Nvidia’s market cap in a single day: the market repricing the model layer as commoditizable. Like power plants and the grid: generation gets cheap and interchangeable, and value migrates to the grid, the transformers, breakers, meters and load balancing that keep the lights on. Routing easy queries to cheap models and hard ones to frontier models, plus caching, routinely cuts serving cost by multiples. You can only do that if routing lives in your layer.

Enterprise model share flipped violently: per Menlo Ventures, OpenAI went from 50% of enterprise LLM API usage in 2023 to 27% in 2025, with Anthropic at 40% and Google at 21%. Teams that owned their evals and abstraction layer switched in days. Teams welded to one API rebuilt. Which one are you?

LangChain’s 2025 survey of 1,340 teams found quality, not capability, is the #1 blocker to production (32%), and 89% of teams now run observability. Evals and guardrails are what turn a demo into a product. And the data flywheel, where every correction and escalation lands somewhere that improves retrieval, evals, or fine-tuning, is the one asset your model provider can never ship to your competitor. MIT’s “learning gap” (tools that do not retain context or improve with feedback) was precisely the #1 pilot killer.

Read that title again without the stress on one word: sometimes the honest answer is just integrate the model and stop there. Not every problem needs the full system, and building one you do not need is its own kind of failure. If some of you are thinking “we really don’t need all that machinery,” you are often right.

The data agrees. Menlo found that 84% of enterprise “agents” are really just a fixed sequence of steps wrapped around a single model call, not the autonomous systems the label implies, and for most jobs that is exactly the right amount of engineering. Remember the reliability maths from earlier: every extra component is one more thing that can break, drift, or need maintenance. Over-building is the mirror image of under-building, and it fails just as quietly.

There is a second reason to stay humble about how much you build: the line keeps moving in the model’s favour. Things you had to hand-build in 2024 became free platform features by 2026. Long context windows swallowed a lot of do-it-yourself retrieval. Built-in web search, memory, tool use and structured outputs replaced code teams used to write themselves. Whatever clever plumbing you build today, assume the providers may ship a checkbox for it next year. And 76% of enterprise use cases are now bought off the shelf rather than built in-house, up from 53% in 2024.

So the rule is not “build everything.” It is: build only the parts the model provider can never own for you. Here is what that phrase actually means. Your foundation-model vendor can sell the very same model to your competitor tomorrow, so anything that lives purely inside the model is not a real advantage. What they can never hand your rival is: your proprietary data, your expertise and knowledge, the evals that encode your specific definition of “good”, the guardrails that fit your domain and compliance rules, and your cost and routing economics. Build those. Rent the raw intelligence, buy the commodity workflows, and design only the parts that actually make you different.

  1. If the model API disappeared tomorrow, what would we still own? ⇒ If the answer is “a prompt library,” you have a demo, not an asset.

  2. Can we swap the model in a week? ⇒ Run your eval suite against two providers this quarter. If you can’t, that is the project.

  3. Where do our evals live? ⇒ In your own repository, version-controlled, and run automatically on every change. Only about half of teams run offline evals before shipping; be in the half that does, because you cannot manage a quality you never measure.

  4. What is our per-step reliability budget? ⇒ Count the steps, do the 0.95^n math, and decide where verification and human gates go.

  5. Who owns routing and cost? ⇒ If every request hits the frontier model, you are overpaying by multiples, by design.

  6. Is the flywheel turning? ⇒ Name the place where yesterday’s corrections made today’s system better. No answer means the learning gap has you.

Come back to the engine we opened with. A Formula 1 team almost never wins on engine alone, and rival teams often buy the same engine from the same supplier. The race is won by the car built around it: the aerodynamics, the tyre strategy, the pit crew, the telemetry read lap after lap. Buying a stronger engine every season (the providers will gladly sell you one) does not make you a winning team. The system around the model is your car.

Yes. The recap:

  • The model is becoming the commodity component ⇒ the price of a given level of capability keeps falling fast (even as each brand-new frontier model launches pricier than the last, so “cheaper” and “more expensive” are both true at once), and market share flips between providers in months. Either way the model is a supplier you swap, not a moat you own; the durable value sits in the system around it.

  • Even “model” records are really system results ⇒ AlphaCode 2 and Gemini’s headline scores did not come from a smarter model, but from orchestration around it (generating many answers, then filtering and voting). The press credited the model; the win belonged to the system. Your production quality will be the same kind of result.

  • Reliability is architecture, not prompting ⇒ 95% per step is 36% over 20 steps; verification and rollback are where the nines come from.

  • The 95%-failed-pilots stat is a system indictment ⇒ pilots die at integration, workflow fit, and missing feedback loops, not model IQ.

  • Simplicity still wins where it can ⇒ 84% of “agents” are workflows, and that is often correct; build only what the model can never own.

Before the one-liner, the concrete version, because “design a system” should never stay abstract. For anything past a prototype, the minimum scaffolding worth owning is roughly this: an eval suite that encodes your quality bar and runs on every change; an abstraction and routing layer so you can send easy work to a cheap model, hard work to a frontier one, and swap providers without a rewrite; guardrails on inputs and outputs for safety, format and compliance; a reliability budget that decides where verification, retries, rollback and human sign-off live; and a data flywheel that turns every correction into tomorrow’s improvement. None of that is exotic, and it is exactly the difference between a demo and a product.

The one-line takeaway: stop asking “which model?” first; ask “are we designing a system or renting a brain?”, and make sure the parts you own, your evals, your data, your guardrails, your routing, are the parts that compound.

Ask yourself the 1st question on your current AI project: if the API vanished tomorrow, what would you still own?

Thanks for reading Artificial Impact! This post is public so feel free to share it.

Share

Research

Practitioner guides

Industry data / news

Read the original on artificialimpact.substack.com ↗

Comments

Nothing yet. Say the first thing.

    Sign in to join the conversation.