I Got Tired of AI Code Review Noise, So I Built a Ratchet

LLM reviewers can flag anything. Only deterministic checks get to block.

For months I ran four LLM reviewers on every diff that mattered: code, security, library, and an adversarial claims reviewer. Four cold-context specialists, every change. It felt rigorous, so I didn't look too hard at what they were actually producing.

When I finally did, the rigor fell apart. The reviewers were flagging things faster than I could work through them, and most of it was not new. It was the same finding from last week, worded a little differently, raised again by an agent that had no idea it had raised it before. I was paying for the same review over and over.

If your review process never turns its recurring findings into deterministic checks, it is not a process. It is noise on a loop. Let an LLM reviewer flag anything it wants. Only a deterministic check should be allowed to block a merge. Most teams wiring up AI code review right now have not separated those two things. I hadn't either.

A human reviewer who flags the same problem three times eventually does something about it. Writes a lint rule. Updates the style guide. Says something in standup that becomes team lore. The third time costs less than the first because people remember, and remembering carries consequences.

An LLM reviewer remembers nothing. Every run starts cold. The agent that flagged a missing validation check on Tuesday flags it again on Thursday with the same confidence and no idea it is repeating itself. My cost per finding stayed flat while the value of each finding fell toward zero. That is not review. It is a subscription to my own backlog.

Volume made it worse. Four reviewers, ten-ish findings each, every day. I became the bottleneck, reading output I had stopped trusting and re-deciding things I was fairly sure I had already decided but could not prove. The false positives came back just as reliably as the real findings, with nothing to tell them apart.

The obvious fix is memory. Save the findings, feed them back to the reviewer next time, let it skip what it already said. I thought about it and dropped it, for two reasons.

The reviewers run cold on purpose. The value of a second opinion is that it has not seen the first one. Prime a reviewer with its own history and it starts agreeing with its past self instead of reading the diff.

And memory solves the wrong problem anyway. A finding that keeps coming back does not need to be remembered. It needs to be dealt with. An agent that remembers flagging the same thing three times is just an agent that remembers being ignored. The missing piece was never memory. It was consequence.

There are two standard ways to handle this, and both make it worse.

The first is to make the reviewer a gate. If the LLM says stop, the merge stops. Now non-deterministic judgment has veto power over your pipeline. The same diff passes Monday and fails Tuesday because the model worried about something different that time. Engineers figure this out fast, and a gate they cannot predict is a gate they stop respecting. The distrust then spreads to everything else the agents touch. That is vibes with veto power.

The second is to make everything advisory. The reviewer comments, nobody is blocked, work continues. It feels safer and rots just as fast, because a finding with no consequence teaches everyone to scroll past it. Give it a few weeks and the advisory output is wallpaper. You are paying for tokens nobody reads.

A finding that recurs without consequence trains you to ignore the reviewer.

Neither failure is the model's fault. The models review fine. The problem is that neither setup separates flagging a problem from blocking on it. The gate fuses them, so every flag becomes a verdict. The advisory split cuts the wire entirely, so nothing a flag says is ever enforced. What you want is the two held apart, with a deliberate path from one to the other.

So I built that path. It shipped today as agentic-os v1.1.0. The core is three tiers, ordered by how reproducible each one is.

Tier 0 is deterministic validators. Scripts, linters, schema checks, grep rules. Anything that returns the same answer every time. This is the only tier allowed to block a merge. If a check can flake, it does not get to gate.

Tier 1 is LLM judgment with evidence attached. A reviewer can push a finding up to this tier only by bringing a deterministic artifact: a failing script, a counterexample, something that exits non-zero on its own. The judgment finds the problem. The artifact is what actually gates. The argument around it does not.

Tier 2 is everything else the reviewer thinks. Style, unease, "this feels wrong." Advisory, never blocking. The part that keeps Tier 2 from being pure noise is that every finding here gets recorded, fingerprinted, and counted.

All four reviewers were rewired to this in the same release. A stop verdict standing on Tier 2 alone is no longer a verdict. It is a flag, and it goes to a ledger instead of to the merge button.

The ledger is where it starts to compound. It is also where the memory lives, outside the reviewers where it cannot pollute a cold read, and attached to a consequence.

It is boring on purpose. An append-only JSONL file, one line per event, driven by a small Python script with five commands: add, tally, triage, promote, retire. Every unevidenced finding gets a SHA-256 fingerprint built from the file path and the normalized claim, so the same defect written two different ways across two runs lands on one entry instead of looking like two discoveries.

{"fingerprint":"a3f29c41d7b08e55","file":"src/api/sessions.ts","claim":"session token compared without constant-time check","tier":2,"source":"security-reviewer","run_id":"r-0611","date":"2026-06-11","evidence":null,"status":"RECURRING"}

Recurrence counts distinct runs, not raw sightings, so a reviewer repeating itself five times in one run cannot fake a trend. When a fingerprint crosses the threshold, triage surfaces it for a human to look at. If it is real, you encode it as a Tier 0 validator or a Tier 1 evidence script, and promote records it. Promote refuses to mark anything done unless the encoded check is attached. The promotion is the check. Findings nobody ever repeats age out through retire.

That is the ratchet: finding, ledger, tally, promote, check. Once a defect class is promoted, no LLM argues about it again. It is out of the stochastic layer for good, and the reviewers go back to looking at what is actually new in the diff.

Once a defect class is promoted, no LLM re-litigates it. The ratchet only turns one way.

Is any of this load-bearing yet? It caught its own first bug before release. The fingerprint normalizer was treating apostrophes in contractions as quote characters, which made unrelated findings collide into one entry. That fix shipped with a regression test in the same release. So did 33 routing collisions across the library, each one a recurring finding that got investigated and encoded instead of re-flagged forever. The system chewing on its own output is the whole idea.

Fair objection. If every good finding ends up as a deterministic check, have I just rebuilt my lint config the long way and thrown out the LLM judgment that was the point?

No, because of the pipeline. Lint rules show up when a human gets annoyed enough to write one. There has never been a standing path from "the reviewer keeps mentioning this" to "the machine checks this now." The ratchet is that path. It does not get rid of judgment. It retires the judgments you have already settled, so the expensive stochastic layer stays aimed at the things you have not. The reviewer stops re-finding what you already know and starts finding the next thing worth promoting. Lint never had that. That is the new part.

Stand up AI code review without this and you get one of the two decays: a gate your team learns to distrust, or comments your team learns to skip. Either way your spend grows in a straight line and compounds nothing. Every dollar buys the same findings the last dollar did.

With the ratchet, the curve bends. Every promoted finding is a review you never pay for again. Quality stops resetting to zero each run and starts accumulating, the way the rest of your tooling already does.

The whole thing, tiers and ledger and rewired reviewers, is open source in agentic-os, one command to install. If the flag-versus-block split named something you have been feeling but had not put words to, star the repo. And subscribe below, where I write up what this system teaches me, usually by going wrong first.

Subscribe on Substack

— Glenn Eggleton builds agentic engineering systems and writes about what survives contact with production.

<details>

<summary>SEO meta</summary>

• Title (≤60 chars): I Got Tired of AI Code Review Noise, So I Built a Ratchet

• Meta description (≤155 chars): My AI code reviewers kept re-discovering the same findings every run. So I built a tier-and-ratchet mechanism that makes review quality compound.

• URL slug: ai-code-review-noise-ratchet

• Primary keyword: AI code review

• Secondary keywords: LLM code review false positives, AI code review workflow

• Search intent: informational

</details>