Most LLM pipelines that look fine in demos are silently failing in production. Here’s a scenario that comes up more than you’d think in real interviews — and even more in production systems.
You’re a Lead Engineer at a fintech startup. Your team receives thousands of raw consumer complaint narratives every day — messy, emotional, unstructured text. Your job is to build a pipeline that turns those complaints into clean, SQL-compatible records.
The downstream database is unforgiving. One bad field type and the record is rejected.
Here’s your target schema:
summary (string) — A neutral, 1-sentence summary of the issue
company_name (string) — Name of the bank or lender. “Unknown” if missing
financial_loss (float) — Dollar amount mentioned, e.g. 50.00. Default to 0.00 if none
is_urgent (boolean) — true if the user mentions legal action, homelessness, or severe distress; otherwise false
The question: How do you guarantee over 90% valid output structure at high volume — when an LLM is doing the extraction?
⏳ How would you solve this? 💭
LLMs are powerful, but they’re not deterministic. Ask one to extract a float and it might return “$50”, “fifty dollars”, or confidently hallucinate a number that wasn’t there. At low volume, you catch these manually. At thousands of records per day, you can’t.
You need a structural guarantee — not just a smart prompt.
Ask most engineers this question and they’ll immediately say: “I’ll prompt engineer it.”
Write a detailed system prompt. Tell the LLM to return valid JSON. Give it examples. Say please, basically.
And honestly? That gets you pretty far in a demo. But in production, prompt engineering alone is begging the model to comply. You’re relying on the LLM’s good behavior rather than enforcing structure. At low volume you might not notice. At thousands of records a day, even a 5% non-compliance rate means hundreds of rejected records — and that’s before the model has a bad day with an unusual complaint narrative.
Prompting is a starting point, not a guarantee. The moment your pipeline’s reliability depends on the LLM “feeling cooperative,” you have a fragile system.
So what does a real structural guarantee look like? Two strategies dominate this problem — and they represent fundamentally different philosophies.
Instead of generating free-form text and hoping it’s valid JSON, you constrain the Instead of hoping the LLM outputs valid JSON, you enforce it at the token generation level — before sampling even happens.
You likely know that Top K and Top P narrow the model’s token probability distribution at each step. Constrained decoding goes one step further: it applies a schema-driven mask that zeroes out any token that would violate your structure. Not unlikely — zeroed out. Impossible to sample.
Example: schema expects financial_loss as a float. After the model outputs "financial_loss":, only digit tokens (0–9) are unmasked. "$50" or "fifty" can’t be generated — the mask eliminates them before the model even gets to choose.
Tools like Instructor, Outlines, or OpenAI’s native response_format implement this masking layer, enforcing your Pydantic or JSON schema throughout generation.
☑️ Reliability: Very high. Schema violations become nearly impossible by design.
☑️ Latency: Low — single LLM call, no retries needed.
☑️ Complexity: Low-to-medium. Define your Pydantic or JSON schema upfront and integrate a structured output library.
⚠️ The catch: Some nuance can be lost when the model is over-constrained. Schema changes require updating the enforcement layer too.
Here you let the LLM generate freely, then run the output through a validation layer — Pydantic, a custom parser — that checks every field against your schema. If validation fails, you retry automatically, feeding the error back to the model so it can self-correct.
This mirrors how humans work: draft, review, fix. It’s flexible and easy to debug because you can log exactly what failed and why.
⚠️ Reliability: High, but probabilistic — depends on retry budget and error feedback quality.
⚠️ Latency: Higher — worst case is 2 to 3 LLM calls per record. At scale, this adds up fast.
⚠️ Complexity: Medium-to-high. You need a validation layer, retry logic, fallback handling, and monitoring.
☑️ The catch: More flexible and far easier to debug — failures are explicit and loggable.
In production, you don’t pick one — you layer both.
Use constrained decoding as your first-pass structural guarantee. Wrap it with a lightweight validation layer that catches semantic failures constrained decoding can’t prevent — like a financial_loss of 9999999.00 that’s technically a valid float but clearly a hallucination. Log every failure. Set alert thresholds. Keep a human-review queue for records that fail after retries.
The goal isn’t perfection from the LLM. The goal is a system that is reliable even when the model isn’t.
📖 This scenario is drawn from The Agentic AI Book — a production-first guide to building AI systems that actually work.
Grab early access: book.ryanrad.org
Until next dose — Dr. Ryan Rad
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.