My AI Code Reviews Ran 20 Rounds. Most of Them Had Nothing To Do With the Code.

A few months ago, I decided to put more emphasis on automated code reviews in GZA, my personal AI coding orchestrator and/or software factory, to increase my confidence in the quality of AI-generated code.

I previously wrote about my motivation for building GZA here:

The Productivity Ceiling of AI Coding Tools

·

Jan 20

Intro I’m sitting at my laptop at 11pm, running Cursor with 4 separate Claude Code conversations happening. I feel more productive than I’ve ever been. But I’d like to go to sleep soon, and unless I keep sitting here, actively prompting and reviewing, no work gets done.

What followed re-taught me something I’ve learned and forgotten many times: you cannot fully automate the understanding of complex systems.

To start taking AI code reviews more seriously, I did 2 things:

1. Upgraded the model I was using for reviews. At the time, I was using Claude Sonnet for coding and Haiku for reviews. I upgraded the review model to Opus, as I believed that creating a list of bugs to be fixed and/or improvements to be made was a more complex task than implementing them. To paraphrase as a question, which is harder: writing a detailed TODO list, or doing the work in the list?

2. Implemented a GZA feature to run automated code reviews and improvements for up to N iterations. We either achieve a passing code review in N or fewer loops and are ready to merge, or, if we can’t get there, we stop and wait for human intervention.

For further clarity, let’s lay out some terminology:

• A task is a unit of work in GZA, with a prompt and a type. Example types are implement, review, and improve.

• A merge unit is a collection of one or more implement/review/improve tasks that result in a mergeable branch.

• A review task (prompt here) performs a code review on work done by a previous implement or improve task.

• An improve task (prompt here) fixes problems found by a previous review.

• An iteration is a pair of coding and review tasks — either the initial implement/review or a subsequent improve/review.

Back to the story. After implementing these changes, I noticed that the number of iterations required to get a passing code review was much higher than I expected:

Over the course of 2 weeks:

• p50 iteration count went from 2 to 3. Not too bad.

• p25 went from 1 to 3. Could be worse, but 75% of my tasks requiring 3 or more iterations feels high to me. Keep in mind, iterations are not free — they all use tokens, and, since I usually limit concurrent tasks to 4, they also occupy system resources that could be used to move other work forward.

• p75 went from 5 to 9. That’s bad.

• p90 and max also increased, but I’m OK with that.

Why did iterations increase?

My first theory was that using different models for implementation and review wasn’t tenable. If the reviewer is smarter, they will always notice things that the implementer misses, and the communication between them will be lossy.

I have no idea why iteration count decreased during the week of 4/14-4/20, and based on subsequent results, it wasn’t worth investigating.

I decided to rollback my review model to match my implement model. By this time, I had switched to using Codex for most background coding work, so I was using GPT-5.3 for implementation and GPT-5.4 for reviews. I switched reviews to GPT-5.3 so they matched.

I ran this way for a week, and the result was clear: it made zero difference. Even when using GPT-5.3 for both implementation and reviews, iterations was still high:

Here’s a scatter plot to illustrate the similarity in behavior across the 2 implement/review model pairs:

I had no obvious answer for how to reduce iteration count. I could have explored ways to improve the prompts I use for both reviews and improve tasks, but I had already done that recently and wanted to let it breathe for a while.

I wondered: was there a difference in the complexity of tasks that required more reviews? My only signal for task complexity is lines of code, so let’s take a look:

At a glance, yes — the lines of code for a given task do seem to scale upward with review iterations.

But I wondered if I had the correlation and causation backwards — if tasks with more iterations had higher LOC since there were more opportunities to write code in each improve task that addressed review findings. I didn’t have concrete enough data on this to create graphs — but from what I saw, this wasn’t the case.

I had evaluated a bunch of data, but still didn’t understand why iterations were so high.

I took a step back and focused on getting GZA 0.5.0 completed, which involved a lot of other rabbit holes that I’ll discuss in future posts.

For now, let’s look at the data I collected for the next 2 months.

The numbers I mention here are from aggregates of aggregates that I compute alongside the graphs.

• p50=2 is good

• p75=3 is reasonable

• p90=6 is higher than desired, but I can live with it

• max=20+ is insane, but that’s the nature of a max.

Confirms the obvious assumption that more complex changes require more review iterations.

It’s also a cool-looking graph.

From the GZA codebase, split between source, tests, and integration tests.

There are a lot of terrible tests in this codebase that should be deleted, but that’s a different post.

p50=~200 seems pretty small, but it’s possible that this graph is just a proxy for the distribution of features versus bug fixes. Generally speaking, features require more code than bug fixes.

When aggregate metrics don’t provide answers, it’s often useful to evaluate specific examples.

So I sampled 5 merge units with high iteration counts and examined the details. Let’s take a look.

• Iterations: 21

• Summary

  • GZA sanitizes prompts before sending them to LLMs, as there are some words considered dangerous (such as “kill”) that can be rejected, even if you’re talking about killing a process, not a living being.

  • The sanitizer rewrote a diff, causing incorrect code to be reviewed, which was then rejected.

  • Review/improve tasks looped infinitely until a human intervened.

• Iterations: 14

• Summary

  • A pre-existing test failure was experienced when verifying a documentation-only change

  • This resulted in a weird mix of changes sent to the reviewer: a doc change, plus a test parallelism change intended to make the test suite run faster and avoid a timeout.

  • The reviewer rejected the change as the test fix has nothing to do with the original intent of the task prompt

  • Review/improve tasks looped infinitely until a human intervened.

• Iterations: 12

• Summary

  • A task was implemented and a review was approved.

  • Multiple tasks were ahead in the queue and were merged.

  • For each merged task, we rebased the task branch in question

  • Each rebase triggered a new review

  • A human intervened and manually merged the task, for some reason

  • After the task was merged, the system couldn’t tell that it was merged and kept trying to review it

  • The reviews had empty diffs since the task was already merged

  • A human intervened (again), to resolve the issue.

• Iterations: 13

• Summary

  • The test runner (a deterministic step that happens in-between agent invocations for each task) saw the test suite failing

  • The subsequent improve task saw the test suite succeed

  • The difference was that the runner was running the suite with per-test timeouts, and the improve task was not

  • They argued for 12 rounds, until the 13th review randomly decided that the improve task’s test results were authoritative (the opposite of the default instructions), marked the review as “approved”, and the task was merged

• Iterations: 21

• Summary

  • This is a counter-example: nearly every round was legitimate.

  • A fix had to be applied to many separate entry points for background task creation (gza retry -b, gza improve -b, etc.), and each review round caught another one that had been missed.

  • ~15 consecutive rounds, each produced a real code fix plus targeted tests. The reviewer was doing its job.

AI is a bazooka. It doesn’t feel safe or correct to ask the bazooka to find its own targets. We’re responsible for identifying the targets and aiming appropriately in order to maximize the probability of success and minimize harm.

Harm in AI coding systems can manifest as malfunctioning code, but you likely already have systems and tools in place to prevent that, such as:

• Automated and manual tests

• System and business metrics

• Canaries and/or gradual rollouts that abort upon metric degradations

What you are less likely to have in place are tools to prevent the much more subtle types of harm that I’ve described here, such as:

• Malformed diffs that corrupt all code reviews and downstream decisions, such as merges

• Unrelated intermittent test failures causing unexpected combinations of changes which fail code reviews when cross-referenced with the original intent of a task or feature

• Inability to accurately detect whether code is merged or not

• Incorrect execution of tests or other internal scripts used to verify correct code before merging it

The software that creates the software can have bugs, too. And while “just point an agent at the logs” sounds cool, without human intervention, you’re just pouring gasoline on a fire.

Building GZA has been an adventure, mostly because it’s provided many problems in which the only path forward was for me to sit down, think about the problem, and start digging my way out.

We’re now in the 2nd half of 2026, and it seems we still can’t automate everything. If you enjoy working as an engineer and receiving a paycheck, that’s great news. Have a wonderful day.

If your team is shipping with AI agents faster than anyone can verify the results, that’s a solvable problem. I help engineering leaders build the metrics, guardrails, and review systems that let teams move fast without merging code nobody understands. Let’s talk.

supremeinformatics.com

If you’ve ever rolled back the obvious cause of an outage and watched the system stay broken, you might enjoy:
Push To Prod Or Die Trying: High-Scale Systems, Production Incidents, and Big Tech Chaos