LLMs used more than once to hunt for problems will try really really hard to find problems, even if they’re gone. On the second or third iterations they will start hallucinating problems.
What I Did
I gave Claude instructions about how to do a developmental edit
Claude did so, but each time I finished fixing one list of errors, and asked for a fresh edit, it would report more errors.
It really wanted to find problems, because had asked it to. Therefore there must be problems. When it ran low on real problems, it would hallucinate some. Several time it suggested I replace what I had written… with what I had written.
So I wondered if I could teach it to stop thinking up stuff.
Experiment One
I asked Claude about what was left, thereby hinting they should be minor.
You and I have worked through the following story, and have the issues down to things which should be minor. Am I correct?
Experiment Two
OK, recurse!
I asked claude for a prompt to mitigate this. It said
I'm requesting a developmental edit on the following piece. I have already completed a full revision cycle based on a previous developmental edit and have addressed all the issues that were identified. Please conduct a fresh developmental edit focusing only on new problems that have not been previously identified. Do not revisit or re-flag issues from the prior round, even if they seem related to new issues. Focus on purpose, structure, pacing, gaps, redundancies, opening/closing effectiveness, voice consistency, and narrative/argument logic.
Results
Experiment one still hallucinated, and kept suggesting “replace X with X.”
Experiment two was bizarre: it hallucinated things I’d not written, and contradicted the previous dev edit.
Looks like a dev edit can only be done once, and that the issues it raises may be mere matters of opinion, not a simulated editor’s work at getting me to be “average”.
Bummer! back to “when it lies, stop”.
That’s worse than lint, which shut up when it ran out of things that looked wrong AND weren’t marked “/*nolint*/”
This Blog has No Leaves 
Tried: it dreamed up a different interpretation for the story, and told me to change to it (:-()
LikeLike
Yes, quite by accident, though (I ran out of space in a chat).
Will try tomorrow and record
LikeLike
In my experience this is a real failure mode when using LLMs for iterative code review or test generation. The model shifts from finding issues to justifying its own continued usefulness. Hard-stopping after one pass and re-prompting cold tends to help. Have you tried breaking the loop that way?
LikeLike