When Large Language Models hit the headlines in late 2022, with much speculation about impending Artificial General Intelligence (AGI) and the displacement of hundreds of millions of knowledge workers – including software developers – I naturally felt I needed to wrap my head around this technology.
After some initial “Wow! How is it doing this?” experimentation, the cracks soon started to show. Sessions with GPT-4 often ended in frustration as the LLM, if it could do what I wanted at all, would require lots of time-consuming coaxing and checking and fixing of outputs.
It would routinely “forget” instructions. It would routinely “lie” to me. And it made a lot of mistakes.
But it was still hard to see what was really going on. On first impressions, LLMs seem like magic.
When I played a game of chess with it, though, the tiger in the Magic Eye picture became visible. Once seen, it can’t be unseen.
For a fair few moves, I was again genuinely amazed that it was actually playing chess. Each move seemed reasonable, and reasoned. Sitting at my home office desk, staring at the screen, I was genuinely getting the feeling that there was some kind of mind looking back at me.
Eventually, the game reached a point where I could see mate in three if I sacrificed my queen. And I distinctly remember thinking, “But surely it can see that?”
It took the bait, and it was indeed checkmate in three more moves. Inevitably.
That’s when I saw the tiger. It doesn’t know where the pieces are. It doesn’t understand the rules of chess. And it’s not looking ahead in the way a human or, more exhaustively, a chess program does.
It’s literally matching the pattern in the sequence of moves so far against, presumably, a large corpus of chess game transcripts in its training data, and predicting what move comes next.
Could be a good move. Could be a bad move. It can’t tell the difference. It has no capacity to understand or reason about chess. It recursively matches input patterns to patterns in the model and predicts what token comes next.
And that’s how LLMs do everything. That’s how they summarise annual reports. That’s how they write poetry. And that’s how they write code. Could be good code. Could be bad code. They have no capacity to understand or reason about code. As a source of truth, this makes them too unreliable for any use case where fidelity matters.
In modular software design, there’s a principle for decoupling called “Tell, Don’t Ask”. I’m going to overload that principle and reuse it in this context.
Instead of asking an LLM for information related to the task at hand, we tell it what it needs to know. Models perform (match and predict) more accurately when the data they’re using comes from the real world and not from the model.
When you “talk” to an LLM, your conversation – your prompts, and the model’s replies – all form part of the context that the model is matching on. That includes all the bad chess moves and all the inaccurate summaries and all the bad poetry. And it also includes all the bad code. All the “hallucinated” libraries. All the incorrectly calculated test data. And all that jazz.
In previous posts, I explained why small, specific contexts work better – produce stronger predictions with fewer errors – and we can expand on that principle by making sure the context in each interaction contains a faithful representation of the real world as it pertains to the task: the code as it is right now, the tests as we specify them, the actual test run results, the actual linter output, the findings of our code review, and so on.
(“Ah, but Jason, LLMs are good at code review.” That doesn’t pass the “Brown M&Ms” test, I’m afraid. Don’t believe me? Take an Open Source code base on GitHub, insert unused imports into randomly-selected source files, and ask GPT-5 or Claude to find them. LLMs aren’t linters.)
Ground every interaction in a more reliable truth. Use deterministic sources of information whenever possible.
And when the model tells you it’s raining, go outside and look!
Codemanship's Blog 