The Best AI Systems Are Mostly Not AI

## The Instinct

When you have a powerful LLM, every problem starts to look like a prompt.

Need to figure out what kind of document you’re looking at? Ask the model. Need to find the right data source? Let the agent search for it. Need to decide how to process something? Build a classifier. You’ve got this incredibly capable reasoning engine. Why wouldn’t you use it for everything?

Because it’s slower, less reliable, and harder to debug than the alternative. And the alternative is usually embarrassingly simple.

I’ve spent the last couple of years building multi-LLM orchestration systems at Talos Title, processing legal documents for real estate transactions. I wrote previously about why a pipeline of models that check each other’s work beats a single model flying solo. This piece is about the part I glossed over: the routing layer. The thing that decides what happens to a document before any model touches it. And the lesson it keeps teaching me about when to use AI and when to just... not.

---

## What Routing Actually Looks Like

People hear “DAG-based document routing” and picture something sophisticated. A machine learning classifier assessing document complexity in real time, dynamically selecting processing paths based on learned features, maybe some reinforcement learning optimizing the routing decisions over time.

Here’s what it actually is: a lookup table.

When a document enters our system, we identify what type of instrument it is — a warranty deed, a deed of trust, a judgment, a lien release. Each instrument type maps to a predefined data model that tells the system what to extract and how. The identification step isn’t a classifier or anything fancy. It’s a lookup.

Match the type, load the model, run the prompts. That’s the routing.

Some instrument types need a little more logic. A judgment, for example, gets a simple follow-up prompt: is this about real property or money? The answer determines which extraction path to follow. But most documents follow a straightforward path from title match to data model to extraction.

Not every problem needs a neural network. Some problems need a well-maintained spreadsheet.

---

## The Humans in the Loop

Here’s where it gets interesting: the lookup table isn’t built by AI. It’s built by subject matter experts.

The lookup table isn’t static — it grows over time. When the system encounters a document type it hasn’t seen before, a human expert maps it to the correct data model. From then on, the system knows. The knowledge base is built by people who’ve been reading these documents for years, not by a training pipeline.

Documents that don’t match anything at all degrade gracefully. We extract what we can universally (the date, the recording information, the book and page) and move on. We don’t try to force an unknown document into a model it doesn’t fit. That’s how you get confidently wrong extractions, which I’ve argued before are worse than no extraction at all.

The end user always has the final say on the extracted data. They can see what the system found and change anything that’s wrong. Most corrections aren’t even AI mistakes. They’re OCR issues. A court stamp obscures a middle initial. A scanning artifact turns “Jr” into gibberish. Old handwritten documents with phrases like “doth it be in the year of our lord” don’t throw the LLMs off at all, which is genuinely entertaining. But a smudge on a 1970s photocopy will trip up any system, AI or otherwise.

So the humans aren’t reviewing every AI output looking for mistakes. They’re curating the knowledge base the system operates on and correcting the small percentage of results where the physical document quality made life hard. Human in the loop, but at the layers where human judgment actually matters.

---

## The County Website Lesson

We recently started a POC that crystallized this whole philosophy for me.

The idea: build an AI agent that retrieves information from government websites each of which is different. This is genuinely agentic work. The agent has to trace the chain of title backwards through time, following ownership from one party to the prior one, deciding what to search for next based on what it just found. That part needs AI. The reasoning about what to look for next, the judgment calls about which results are relevant: that’s where an LLM earns its keep.

But the very first thing we tried was letting the LLM find the correct government website. The result: it kept pulling up the SEO-optimized third-party sites that outrank the actual municipal ones in search results. Not even close to reliable.

Our fix: have a subject matter expert define the correct URLs. There are roughly 3,000 counties in the US, which sounds daunting until you remember that our experts visit these sites every day. They already know them. It will probably take less time for a human to define them than we spent trying to get the LLM to find them reliably.

See the pattern? Same architecture as the document pipeline. Humans curate the knowledge. AI operates within it. The lookup table of county websites is the same idea as the lookup table of instrument types. Human-built knowledge that the system depends on, maintained by people who actually know the domain.

The instinct was “let the AI figure it out.” The right answer was “just ask someone who already knows.”

---

## Why This Is a Hard Sell

“Our AI system is mostly a lookup table and some standard code, with LLMs at the specific points where reasoning actually matters” is not a great pitch deck slide. It doesn’t demo well. It doesn’t make for exciting Twitter threads about how AI is disrupting the industry.

“Our AI handles everything end to end” is a much better story. It’s just not a true one, at least not for anyone shipping production systems in domains where accuracy matters.

The benchmark culture reinforces this. Model capability gets measured and ranked. Pipeline design doesn’t have a leaderboard. The press covers model releases, not the routing infrastructure that makes models useful. So there’s a persistent bias toward overusing AI, toward reaching for a model when a simpler tool would do the job better, faster, and more reliably.

I’ve caught myself doing it. The instinct to “just prompt it” is strong when you work with these tools every day. But every time I’ve replaced an LLM call with a lookup table, a simple conditional, or a human-curated knowledge base, the system has gotten more reliable. Not more impressive-sounding. More reliable. Those are different things.

---

## Drawing the Lines

The engineering judgment in building AI systems isn’t about how to use AI. It’s about where to use AI. And more importantly, where not to.

We didn’t optimize our OCR pipeline further even though it’s the source of most correction-worthy errors. The quality is good enough, and the ROI on improvement is better spent elsewhere in the workflow. That’s not laziness. It’s knowing where not to optimize, which is a different skill than knowing how to optimize, and arguably a harder one.

We didn’t build a classifier to route documents. We built a lookup table curated by domain experts. We didn’t train a model to find county websites. We’re going to have a human list them. We didn’t automate the mapping of unknown document titles. We built a UI where experts do it.

The LLMs do what LLMs are genuinely good at: extracting structured data from messy unstructured text, reasoning about ambiguous content, and making judgment calls about what to search for next. Everything else is standard engineering. And that’s not a limitation of our system. It’s the design.

The question isn’t “can AI do this?” It almost always can. Eventually, expensively, unreliably. The question is “should AI do this, or is there a simpler tool that does it better?” The best AI systems are the ones where someone had the discipline to answer that question honestly at every layer of the stack.

---

*Andy Barilla is Director of Language Engineering at Talos Title, where he builds multi-LLM orchestration systems for legal document processing. He has been writing software for over 30 years.*