Most RAG systems that work beautifully in the demo quietly fall apart the moment real-world complexity shows up. The moment you’re dealing with hundreds of dense, overlapping regulatory documents across multiple jurisdictions — the kind of corpus that actually matters. Here’s a scenario that comes up constantly in production — and increasingly in interviews.
THE SETUP
You’re a Senior Engineer at a global law firm operating across four jurisdictions: the US, the EU, the UK, and Canada. Each has its own massive mountain of regulatory rulebooks — chapters, articles, and sub-clauses.
Staff use a live chat interface during client calls to ask high-stakes questions like: “What are our disclosure requirements for retail clients?” or “How long must we retain transaction records?”
A colleague already built a first version. It dumps all regulatory content into a single vector store and retrieves the top-5 most similar chunks for any question. In testing, it has two critical problems:
Hallucination Soup: It blends US and EU rules into one “franken-answer.” Technically well-written, dangerously wrong.
The Lag: Response times have crept past 10 seconds. In a live client call, 10+ seconds is an eternity of awkward silence. Particularly, if you need it more than once!
The task: Fix the accuracy and get the latency under 2 seconds.
⏳⏳ How would you solve this? 💭
⚠️ Why the “NAIVE RAG” is Failing?
The single-vector-store design has two problems, and they share the same root cause: treating all regulatory content as one undifferentiated pile of text.
Semantic Overlap: “Data retention” looks the same in a vector space whether it’s Canadian law or UK law. The retriever returns a top-5 list that’s a geographical smoothie. Vector similarity doesn’t understand jurisdictional boundaries — it just matches surface-level meaning. In a regulated environment, citing the wrong jurisdiction’s rule isn’t a minor bug. It’s a compliance incident.
The “Needle in a Haystack” Tax: Searching 100,000+ chunks is computationally expensive. As the document library grows, latency scales with the data. That’s how you go from 2 seconds in the demo to 10+ seconds in production.
🙋 The Instinct Everyone has FIRST - NOT Wrong, NOT Enough
Most engineers hear this and immediately say: “Just add a metadata filter for jurisdiction!”
Tag every chunk with its country and regulation name. Filter at query time.
That helps accuracy. It’s not wrong. But you’re still doing a massive vector search within that jurisdiction. If the EU regulations alone are 5,000 pages, you’re still hunting for a needle in a slightly smaller haystack. Latency remains high because you’re still running dense embedding comparisons across thousands of chunks.
Metadata filtering is a good ingredient. But it’s not an architecture.
The next instinct is usually: "Fine, I'll add an LLM router — a small model that classifies each question into the right jurisdiction." Better thinking, but still overengineered. You're adding an LLM call, a classifier to maintain, and a fixed set of routing rules that break every time a new regulation is added. There's a simpler way — and it comes from the retrieval system itself.
🌲 The PRO Solution: HIERARCHICAL RETRIEVAL
The core insight: figuring out which jurisdiction and regulation applies is a fundamentally different task from finding the exact clause that answers the question. A flat vector search tries to do both in one step. Hierarchical retrieval separates them into stages — each one fast, each one precise.
To hit sub-2-second latency and jurisdictional accuracy, you need to stop “searching” and start “navigating.”
Phase 1: Document-Level Retrieval (The “Where?”)
Before you search any chunk-level index, search a document summary index first.
Here’s the trick: you pre-generate a one-paragraph summary for each regulation in your corpus. If you have 100 large regulatory documents across four jurisdictions, that’s 100 vectors — total. When a question comes in, you embed it and find the closest document summary. That’s a nearest-neighbor search over 100 vectors. It’s essentially a lookup — sub-millisecond, no LLM call needed, no classifier to maintain, no routing rules to write.
Input: “What are UK disclosure rules?”
Top-1 match: Summary of UK_FCA_Conduct_of_Business_Sourcebook
Action: Drill into that document’s dedicated chunk index.
And it scales naturally — when a new regulation is added, you just generate its summary and add one more vector. No retraining, no rule updates.
Phase 2: Hierarchical Fetch (The “What?”)
Now that you’re inside the right index, don’t search for paragraphs. Search for summaries, then drill down.
Retrieve the Chapter: Search against high-level chapter summaries. Fast — the index is tiny.
Retrieve the Section: Within that chapter, search the specific sections.
Fetch the Chunks: Pull the raw legal text (child chunks) associated with that section and feed them to the LLM.
This is Small-to-Big Retrieval: embed small, punchy summaries for the search phase to keep latency low, but feed the full parent text to the LLM so it has the actual legal context it needs. And because all retrieved chunks come from a single, known source document, traceability is built in — you can require the model to cite the specific article, chapter, and clause. No extra work needed.
🧠 WHAT A SENIOR ENGINEER WOULD ACTUALLY SAY
“In production, retrieval is a multi-stage funnel, not a single search.”
Use hierarchical retrieval as the backbone, then layer in: a re-ranker after Phase 2 to reorder chunks by true relevance (cheap — tens of milliseconds, meaningful accuracy boost). A fallback path for ambiguous queries that span jurisdictions — route broader and flag for review, never guess silently. And aggressive caching — compliance questions are repetitive; a semantic cache on the top 50 queries can skip the entire pipeline for most of your volume.
📖 This scenario is drawn from The Agentic AI Book — a production-first guide to building AI systems that actually work.
Grab early access: book.ryanrad.org
Until next dose — Dr. Ryan Rad
Join engineers and AI practitioners getting smarter about Agentic AI — one DOSE at a time
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.