Four customer calls this quarter. Same problem, four phrasings. The plainest version of the question: once we see the bad traces, what’s the fix? I had an answer that sounded complete and was actually half of one.
The honest half was: build a certification pipeline. I had one, a build I haven’t written about here before: three finance agents (a 10-K analyst, a sentiment triage agent, an advisory drafter) evaluated against FinanceBench and Financial PhraseBank data, all synthetic. Golden datasets, deterministic scorers, an LLM-as-judge for groundedness, a gate that fails the build if any dimension misses. The internal pitch that made it real: certification evidence that takes a model risk team two weeks of manual testing comes out of this pipeline in a day. It worked. It passed. It shipped a green checkmark every time I ran it.
The half I was skipping: a certification gate that only runs when you remember to run it isn’t a gate, it’s a snapshot. Two things kept quietly falling outside its reach: production failures that never became test cases, and prompt changes that never got re-tested. I’d built a pipeline. I hadn’t built one a loop.
This is the build log for closing both gaps.
It’s also where three earlier builds converge, if you want the lineage: the observability layer gave me the traces, quality gates for AI-generated code gave me the gate reflex, and the self-healing CLAUDE.md proved a system can repair its own config. This issue points all three at the eval layer.
Any eval pipeline you build once and run manually rots in two places, and it’s not specific to finance or to Langfuse.
Rot point one: production failures die on arrival. Monitoring catches a bad trace, someone reviews it, maybe fixes the prompt by hand, moves on. The trace that proved something was wrong never becomes a test case. Six months later a different edit reintroduces the same failure, and nothing catches it. You fixed the instance. You didn’t fix the class.
Rot point two: not every deploy goes through your pipeline. If certification only runs on a git push, it has a blind spot the size of every other way a prompt reaches production. Managed prompt platforms let you edit and promote a prompt version directly in the UI, no code involved. That’s a deploy, and it walks straight past a gate watching git instead of production.
Neither is a code bug. They’re structural. A one-shot pipeline is a photograph of the day you built it. The world it’s evaluating keeps moving.
If you’re making the case for continuous eval internally, here’s the framing that works: “A certification gate you have to remember to re-run isn’t a gate. It’s a snapshot with a compliance report attached.”
The pattern underneath this: across deployments I’ve been in the room for this quarter, the recurring question isn’t “which eval platform should we buy,” it’s “how do we build a methodology that survives us switching platforms,” vendor-neutral, because the tool changes and the discipline has to outlast it. The two edges below are that discipline: the feedback paths any eval system needs, regardless of platform.
My first fix was the obvious one: schedule the whole suite to run nightly, no matter what changed.
It solved almost nothing. A bad promotion could sit in production for up to 24 hours before a scheduled run noticed it, and a nightly run against a static dataset doesn’t know a new failure mode exists until someone adds it. It also burned real cost re-testing everything, every night, whether or not anyone had touched anything.
The fix wasn’t “run it more.” It was “run it when something specific happens.” Event-driven, not cadence-driven. Two triggers, two edges, closing the moment the underlying thing changed.
Langfuse names the five-stage backbone the AI Engineering Loop: Trace, Monitor, Build Datasets, Experiment, Evaluate. It’s the right backbone: every request traced, production monitored, datasets built from what you see, experiments comparing one variable at a time, a gate deciding ship or no-ship.
I run two self-hosted Langfuse-on-ClickHouse estates, a personal agent fleet and a work one, so the mechanics below are what I’m actually doing.
What I added is the piece that makes it a loop instead of a five-stage pipeline: two feedback edges that route the loop’s own output back into its own input, with no human remembering to close either one by hand. I’m calling the two-edge closing mechanism the Self-Healing Certification Loop.
Edge A, Observation → Development. A production trace gets flagged, lands in a review queue, and once a human confirms it’s real, it gets promoted into the golden dataset as a new regression test. The next certification run tests against it automatically. A production failure becomes permanent coverage, not a one-time fire drill.
Edge B, Ship → Re-certify. When a prompt gets promoted to production inside Langfuse (a deploy that happens entirely outside git), that promotion fires a webhook, which triggers CI, which re-runs the certification gate against exactly that new version. A regression turns the workflow red before anyone finds out the hard way.
The design constraint that matters more than the wiring: neither edge is allowed to trust its own trigger. Edge A never promotes a flagged trace’s own output as the “correct” answer, because a flagged output is by definition under suspicion. Edge B never trusts the payload it receives about what changed, because payloads can be truncated or stale. Both edges are built to distrust the trigger and go verify. That’s the difference between a loop that heals and a loop that quietly poisons itself.
Rubric drift is still a human problem: the loop keeps coverage current, but does nothing about whether “correct” still means the same thing six months from now. Evaluator quality is a ceiling, not a detail: a loop that faithfully re-runs a bad evaluator just re-runs it faster, with more confidence.
The refusal, restated once more because it’s the whole point: Edge A will not promote a flagged trace’s output as ground truth, even when it’d be more convenient to auto-approve the backlog. Remove that friction and the loop stops closing gaps and starts manufacturing them.
The certification gate you build once is a photograph. The one that heals itself is a habit you never have to remember to keep.
What’s the messiest part of your current eval setup, the thing that only breaks the week nobody’s watching it? Hit reply, I read every one.
Know a builder who would actually run this? Forward it to them, it takes 5 seconds.
👀 Anthropic confidentially filed for an October Nasdaq listing, base-case valuation over $1T (Zacks). If you’re building on Claude, the next four months of pricing and product decisions are pre-IPO positioning, not neutral product choices.
↓ Claude Code shipped obfuscated geo-detection logic fingerprinting Chinese corporate networks, then quietly removed it after a Reddit user reverse-engineered it (Tom’s Hardware). Alibaba banned the tool company-wide. Audit your vendor’s telemetry, don’t assume it doesn’t exist.
↑ A new CLEAR framework (arXiv preprint) found agents tuned purely for accuracy cost 4.4 to 10.8x more than cost-aware alternatives delivering comparable results, across 300 enterprise tasks (arXiv:2511.14136). This loop’s gate measures correctness, not cost per correct answer, worth flagging as an honest gap.
👀 Karpathy’s “agentic engineering” framing is now four consecutive weeks of sustained derivative discussion (amplification via X). The loop above is what agentic engineering looks like applied to eval infrastructure specifically: don’t just write evals, build the loop that keeps them honest.
↑ Langfuse shipped ClickHouse query timeout enforcement at the platform level in v3.203.3 (Langfuse v3.203.3 release notes). Direct relevance here: the stack this loop runs on just got more resilient under the query load a self-healing pipeline generates re-certifying on every prompt promotion.
More Signal. Less Noise.
Get the complete build below: what the certification gate actually measures (the two-level gate and the full evaluator stack), the production monitor that feeds Edge A, the promotion script with its human-gating logic, the full GitHub Actions workflow and its dedup fix, what the loop costs to run, the activation checklist, the “wire this into your own stack” notes, and the public repo. If you want the working code alongside every future build log
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.