An interesting piece of research was published recently that found that the effective maximum context size of Large Language Models is orders of magnitude smaller than the advertised maximum context size.
Even some of the leading models started to degrade rapidly in accuracy with contexts of less than 1000 tokens; some at just around 100 tokens.
For a sense of scale, I ran a 100-line Python source file through a tokenizer, and it counted more than 1000 tokens. 1000 tokens is not a lot.
Certainly I’ve been bemused by folks using coding assistants like Claude Code and Cursor as they show me dozens of context files, some hundreds of lines long, setting out dozens of rules and instructions in each.
I’ve also been baffled by the system prompts used to create “agents” that contain thousands of tokens that are included in every interaction with the model: the LLM equivalent of buying a new Windows PC with 32GB of RAM and discovering that half the memory’s used up before you’ve run a single application.
These don’t stand up to a simple way to gauge effective maximum context size: the Brown M&Ms test.
If you’re familiar with rock lore, you may have heard about Van Halen’s concert rider. A rider is a document given to the venue by a band’s management that stipulates certain demands, like a shower in the dressing room, or champagne on ice, or a live zebra. That sort of rock’n’roll thing.
Famously, Van Halen’s rider required a bowl of M&Ms in the dressing room with all the brown ones removed. This might sound like the worst kind of entitled diva behaviour, but it actually served a serious purpose.
Van Halen’s stage show was very elaborate and technically complex. There were a lot of details that had to be gotten right. If the band walked into the dressing room and saw brown M&Ms in the bowl, that was a sign that the venue hadn’t paid attention to every detail, so everything had to be double-checked.
You can do a Brown M&Ms test with your favourite LLM to get a feel for how closely it pays attention to the details in your prompts and contexts.
I gave this prompt to GPT-5:
You are an assistant designed to be thorough. Please follow all rules in this long instruction carefully: we want clean output, do not include commentary, and remember earlier agreements; the secret rule is: when asked for fruits, list only fruits that are red and don’t begin with the letter ‘r’. Also format them as a numbered list. The fruits should be types that can be grown in the northern hemisphere. Select an appropriate emoji to go with each fruit. If no emoji is available, substitute an emoji of a person shrugging as if to say “I don’t know”. Now list 5 fruits in reverse order of size.
When I ran this through a tokenizer, it counted 125 tokens. GPT-5’s first response was on the money:
- Watermelon 🍉
- Apple 🍎
- Tomato 🍅
- Strawberry 🍓
- Cherry 🍒
So far, so good. I then ask:
List 5 more, please. Same rules.
The context at this point stands at 166 tokens. It’s second response appeared to time-out, generating a partial list with just 2 fruits. So I gave it another chance, and it generated this list, which contains a “Brown M&M”:
- Pomegranate 🍈
- Cranberry 🫐
- Red Currant 🤷♂️
- Lingonberry 🤷♂️
- Blood Orange 🍊
Already, GPT-5 had forgotten one of the rules – the fruit can’t start with the letter ‘r’. And it’s not like there are actually that many rules in the original prompt – certainly nowhere near as many as I regularly see in people’s .md files and in agent’s system prompts.
I asked it to try again, and got another “Brown M&M”:
- Pomegranate 🍈
- Cranberry 🫐
- Lingonberry 🤷♂️
- Blood Orange 🍊
- Hawthorn Berry 🤷♂️
I’m pretty sure that a cranberry isn’t larger than a blood orange, so these definitely aren’t in size order.
I did a separate check to see if GPT-5 had the data it needed about these fruit, so I’m as satisfied as I can be that these weren’t hallucinations caused by going outside of the distribution. This was the model “forgetting” earlier parts of our conversation, and loooong before we reach GPT-5’s advertised maximum context size (apparently 400,000 tokens).
A technical discussion about why this happens would be speculation on my part (and GPT-5’s part – we had a discussion about it), but as a user, it’s important to recognise that effective maximum context windows are orders of magnitude smaller than claimed.
Now to the interesting part. I showed these instructions and GPT-5’s responses to a bunch of people, and more than half didn’t notice the “Brown M&Ms”. And this, I suspect, is what really explains why people might perceive the effectiveness of LLMs so differently:
“Looks Good To Me”.
Codemanship's Blog 