One Day of LLM Agent Observability: Five Production Bugs, Three JSONL Files

I run a small side product: a chat agent that builds real, printable children’s books with kids — Claude Sonnet does the words and the tool calls, a FLUX diffusion model on my RTX 4090 paints the pictures, Typst lays out a print-ready PDF. A handful of families use it. Until this week it had almost no observability: one journald line per model turn, token counts hardcoded to zero, and image generations that left no record at all.

In one morning I gave it a proper trace layer — nothing fancy, three append-only JSONL files — plus evals that run against the production model path. This post is about what happened next, because it’s the strongest case for observability I’ve ever had first-hand: the logs paid for themselves the same day, five times. All numbers below come from the actual trace files, which I’ve kept as a snapshot alongside the analysis script that derives them.

The trace design

The design goal was the smallest thing that answers real debugging questions, with no new infrastructure. Three files, one writer process each, tied together by one shared id (trace — the book’s id, or the draft chat’s key):

<books>/.logs/
  turns.jsonl        # web app: one record per chat turn
  model-calls.jsonl  # LLM proxy: one record per model call
  image-calls.jsonl  # paint pipeline: one record per diffusion run

A model-call record carries what you’d expect from any tracing platform — tokens, cost, latency, finish reason — plus two product-specific flags I’ll get to:

{"at":"2026-07-30T11:52:23Z","trace":"ms7gcs1w-irbbsa","msgs":8,
 "latencyMs":5388,"finishReason":"tool_calls","toolCalls":["illustrate_page"],
 "parseFailure":false,"mixed":false,
 "usage":{"input_tokens":10,"output_tokens":144,"cache_read_input_tokens":21828},
 "costUsd":0.0106}

An image-call record captures the full composed prompt, seed, model, and — the field that earned its keep within hours — paintMs next to total latency, so I can tell time spent waiting for the GPU apart from time spent actually painting.

Two deliberate choices. First, the turn log stores message lengths, never text — these are children’s conversations, and an operational log you can grep freely is worth more than one you have to treat as radioactive. Full transcripts already live with each book; prompt/reply capture for replay exists behind an env flag with a 7-day TTL. Second, no tracing platform yet. jq over JSONL answers everything at this scale, and the records deliberately mirror the shape tools like Langfuse (or any OpenTelemetry GenAI backend) expect — so moving to a platform later means changing where the records are sent, not what they contain. (For the tooling landscape and when a platform is worth it, see my evals and monitoring guide.)

The shared id travels in a request header the chat session stamps on its model client, so one grep links a chat turn to its model calls, its paintings, and the server logs.

Bug 1: 73% of GPU time was habit, not product

First real test: I drove the app end-to-end with a Playwright script playing “parent” — it logs into the real web UI, asks for a coloring book of animals, answers the agent’s questions. (The script knows a turn has finished by watching the new trace log for the turn’s closing record — the logs were dogfooding themselves from minute one.)

The book came out fine. The trace told a different story:

paints: 33 | avg 60s | total 33.2 min | errors: 0
refs: 24  pages: 8  cover: 1

24 of 33 generations — 73% of GPU time — were character reference sheets. Reference sketches exist so a recurring hero looks identical across pages. A coloring book of one-off animals doesn’t need them; the storybook flow was simply being reused out of habit for a different product. One image-calls.jsonl line made that visible and quantified.

The fix was a flow redesign, not an optimization: coloring books became first-class (skip character records for one-off subjects, describe each animal fully in the scene prompt), and the reference-count default became style-aware. Same ask, second drive:

Bug 2: the prompt forbade file paths; the tools kept handing them over

The same drive showed the agent cheerfully telling a child: “Open your PDF here: /home/pedro/plotpilots-dev/books/…”. The system prompt explicitly forbids showing file paths. The prompt wasn’t the problem.

The tool results were. export_pdf returned The book is ready! PDF: <absolute path> — and a model relays what its tools hand it, prompt rule or not. Four tools did some version of this. The fix wasn’t more prompt; it was rewriting tool-result text to say what the reader’s world contains (“it’s on the book screen”) and moving paths into structured fields the UI can use but the model won’t narrate.

The transferable lesson: tool output is prompt engineering. Every string a tool returns is a model input with exactly the same authority as your system prompt — and it’s the input teams audit least. I added a mechanical eval check that fails any transcript containing a filesystem path, so this class of bug can’t ship quietly again.

Bug 3: Spanish books for English kids — and why I stopped prompting

My eval suite — scripted kid requests played through the real agent and tools, graded by mechanical checks and an LLM judge — had always run against a local Qwen model. Production had quietly moved to Sonnet through a proxy. The first time the evals ran against the production setup, the judge flagged something the local model never did: a request written in English produced a book written entirely in Spanish.

I strengthened the prompt (“mirror the child’s language”, “one language per reply, never mix”). It kept happening — 3 of 7 runs wrote the wrong language. Prompt exhortation was a coin flip.

The fix that worked was removing the model’s discretion entirely: the app now sniffs the language of the child’s first real typed words (a 40-character regex — this is a two-language product) and injects it into every turn as ground truth: [app note: the child's language is English — every reply and every word of the book must be in that ONE language]. Evals inject the same note, so they test the production contract. Language failures went from 3/7 to 0 across every run since.

When a behavior must be reliable, stop asking the model to infer state you already know. Detect it, store it, inject it.

Bugs 4 and 5: how models break their own tool protocol

The proxy translates between OpenAI-style tool calls and a text protocol (“reply with ONLY this JSON object…”). Text protocols are conventions, not contracts, and the traces caught Sonnet breaking the convention in two directions within hours of each other:

Prose, then JSON. All 10 pages are painted! 🎉 {"tool_calls": [{"name": "make_cover"… — and because the streaming guard only checked whether a reply started with a brace, the raw JSON streamed straight into the chat bubble. The fix is a stream holdback: any mid-prose { is withheld until it either diverges from the tool-call prefix (released — it was innocent prose) or matches it (silenced and parsed). I regression-tested it by swapping the real model CLI for a fake that replays the exact failure.

Code inside JSON. One run died at zero pages because the model emitted "bookId": "tina-...-7t3j".replace("turtle","volar") — JavaScript inside what must be literal JSON. Unparseable, so the tool call silently became chat text and the agent run ended. The proxy now rejects unparseable tool-call attempts with a real HTTP error (the app shows its friendly retry bubble), instead of letting raw JSON reach the reader.

Both failure modes are now counted, not just fixed: every model call logs parseFailure and mixed booleans. If a future model update regresses the protocol, that’s a number moving on a log I can alert on — not a confused kid.

Eval history that lives in git

The last piece cost twenty lines: every eval run appends one line per scored case to a committed history.jsonl —

{"at":"2026-07-30T13-16-35","sha":"e96f3ee","model":"claude-proxy/claude-proxy-sonnet",
 "id":"dragon-candles","mechanical":true,"overall":2,"language_ok":false}
{"at":"2026-07-30T14-05-12","sha":"052bd68","model":"claude-proxy/claude-proxy-sonnet",
 "id":"dragon-candles","mechanical":true,"overall":5,"language_ok":true}

— so every score is attributable to the commit and model that produced it, regressions show up as diffs, and jq gives you trend lines without a dashboard. For a solo product or a small team, this is embarrassingly effective. It also caught a judge bug the same day: the judge was reading each page’s raw layout setting instead of how the page actually prints, and penalizing coloring books for a uniformity that is the format working as designed. Judges need the same product context your users have.

The payoff, live

The third drive of the day put the pieces together. I’d just added a “sign the book” step — author byline and an optional dedication — built the same way as the language fix: not as a prompt plea (“remember to ask their name!”) but as state. The byline and dedication are fields on the book, and after every action the app tells the agent what the book still lacks; the persona only supplies the warmth. Requirements gathering in an agent product is state, not vibes — a prompt line gets forgotten; a reminder computed from what’s actually missing cannot.

After painting all ten pages, the agent — unprompted by the driver — asked its two questions, and the finished PDF carries a byline (“Teo & PlotPilot”) and this, printed on the page facing the title:

And mid-run, the observability earned one more keep: the mixed counter logged exactly one prose-then-JSON reply. The stream guard held it back — the chat stayed clean — and the log recorded that it happened. That’s the end state you want: known failure modes that are counted when they occur and invisible to users when they do.

What generalizes

This was one day on a small product, but the checklist is the same one I set up for client engagements:

1. One shared id across every log, before anything else. A log line you can’t connect back to a session answers nothing.
2. Audit tool output like you audit prompts. It’s the same channel.
3. Detect and store what must be reliable instead of asking the model to remember it: language, user identity, requirements gathered mid-conversation.
4. Count protocol violations as metrics, don’t just patch them.
5. Test the exact setup you ship. For weeks my evals were validating a model I wasn’t even running in production; the day they aligned, they found real bugs immediately.
6. Commit your eval scores with the commit and model that produced them.

Honesty notes: the product and infrastructure are mine, the logs are real snapshots (kept with an analysis script that re-derives every number above), the child in the test conversations is fictional, and the dollar figures are API-equivalent costs computed from the token usage the model CLI reports — the words actually run through a subscription while the beta is friends-and-family.

Update, one day later

The pattern kept compounding, so three additions worth recording. First, the three files became four: the paint pipeline now has a vision critic. Every generated page gets one look — does the picture show the scene? is there exactly one hero? any fused anatomy? — and at most one guided repair, with the verdicts landing in critic.jsonl under the same shared id. On its first production day it repaired one page in ten and lifted the scene-accuracy score from 3.55 to 3.89, for about 10% more GPU time. The failure modes aren’t just counted anymore; they’re corrected, and the corrections are counted too.

Second, the “judges need product context” lesson above turned out to be the small version. I asked the agent for an adult gift book; the writing came back genuinely witty, and my children’s-book-editor judge scored it 2 out of 5 — for being a bad kids’ book, which it wasn’t trying to be. The same manuscript under an adult-gift-book judge: 4 out of 5. A miscalibrated judge doesn’t just add noise; it fails good work for the wrong reason, and every decision you base on the score inherits the error.

Third — and this is the one that changed how I think about the stack — a judge complaint drove a product redesign, not a bugfix. My “picky parent in a bookstore” judge, looking at the actual printed pages, wrote: “half the book is blank pages… feels like padding, not a finished €25 gift book.” It was right: a ten-page story printed one page per sheet fills only 14 of a print-on-demand book’s 24-page minimum, and the rest gets padded with blanks. The fix was a format change — proper picture-book page pairs, words on the left and a full-page picture on the right, so the story fills its own book — verified the same afternoon: blank pages down from ten to three, and the judge’s coherence score doubled. The full experiment program behind that — six experiments in a night, and why I then closed the lab — is its own post.

If you’re shipping an LLM feature and can’t currently answer “what exactly did the model see when it did that?” — that’s usually a one-day fix, and in my experience it starts paying back the same afternoon. I do this for clients: work with me.