Bigger Windows Aren’t the Answer

In Part 1, we looked at how the AI models that we use don’t actually “remember” anything. We learned what AI context is (the entire conversation) and how the context window is generally a very limited amount of space to work with.

We also discussed the two main approaches that the industry uses to deal with more information and longer conversations flowing into the LLM:

1. Compaction: summarizing old messages to make room for new ones

2. Bigger context windows: giving the model more space to work with

Both are brute-force solutions with substantial costs. We’ve heavily discussed compaction, which causes the LLM to lose a massive amount of detail. Bigger context windows sound like the perfect solution, but they are costly both in terms of money and the impact to the environment. But let’s say we could somehow solve the cost problem and become very energy efficient, we should be good, right? Ehhh, not so much!

Most of the big companies offer models with massive context windows today. OpenAI’s GPT-5 and Google Gemini advertise context windows of over a million tokens (about the size of the entire Harry Potter series) being sent for every prompt. But what happens when we actually use all that space to put in a bunch of information?

If I gave you one day to read as many books as possible, would you be able to quote me a single sentence from any of those books at the end of the day? I doubt it. Something similar happens to LLMs with these huge context windows. They accept the huge data but they aren’t really able to use it. There is a way to see this problem in numbers, since AI models are tested against a benchmark. In this case, they are being tested on behavioral performance in terms of quality of the answers.

One such test is called “MRCR v2” and it’s trying to determine how well a model performs if you throw multiple needles (details) in a haystack (context window) and require the LLM to reason its way into of finding them. So, if I asked you, “What animated images were on the plates in Umbridge’s office?” after reading the Potter series, would you remember that fact? Even AI LLMs often don’t.

Against this test, Google’s own evaluation card showed Gemini 3 Pro scoring 77% in a context window of 128K tokens but dropping to just 26.3% at the actual 1M-token mark. That’s a massive degradation, nearly a 3x performance drop when you actually fill the window.

So, clearly, just having a large window isn’t especially useful. In fact, interestingly enough, massive context windows often end up distracting the model from answering accurately. So now we have three problems:

What if the answer isn’t a bigger window or a better summary? What if it’s a more relevant window?

This is where a new idea towards context management comes into play: a project called “virtual-context”.

LongMemEval is a benchmark published at ICLR 2025 designed to test long-term memory in multi-session conversations. Each question requires recalling specific facts, temporal relationships, or knowledge updates scattered across a haystack of roughly 118,000 tokens spanning dozens of sessions. The questions span six categories: single-session recall (user, assistant, and preference), multi-session recall, temporal reasoning, and the hardest category: knowledge-update questions, where a fact changes across sessions and the model has to figure out which version is current.

I ran 100 randomly sampled questions from the full 500-question benchmark. The baseline: Claude Sonnet 4.5 (a mid-tier model) with the full haystack, every token, every session, all ~118,000 tokens in the context window.

The test: the same model, enhanced by virtual-context to manage what it sees.

Without virtual-context, the model got 33 out of 100 right. With virtual-context, the same model got 95 out of 100, while using 55% fewer tokens at a 55% lower cost.

Multi-session and knowledge-update (the questions that require locating and connecting facts scattered across different sessions) are where the baseline collapses. But those are exactly the situations where the real-world use case of LLMs apply. When’s the last time you had a single-session, single-topic interaction with AI? The improvement isn’t marginal; there are significant jumps in success rate. 33% to a highly reliable 95% isn’t peanuts.

The baseline had all 118,000 tokens available for every question with every fact, every session date, every detail, right there in the window. And yet the model still failed on 67% of the questions. The information wasn’t missing or unavailable to the model. This is what researchers call context rot. As the context window fills up, the model’s ability to locate, prioritize, and reason over specific facts degrades, even though those facts are technically present. All of the details are there; the model just can’t effectively use them anymore because they’re surrounded by tens of thousands of tokens of other information competing for attention.

This is most easily seen in the knowledge-update category. A user says “I keep my sneakers under my bed” in a session on May 25th. Four days later, in a different session, they say “I moved my sneakers to the shoe rack in the closet.” The session dates are right there in the conversation. Now we ask: where does the user currently store their sneakers? The baseline gets it wrong. Not because it can’t see the dates, but because it can’t reliably navigate 118,000 tokens to find the two specific statements, correctly interpret which session came later, and prioritize the more recent one. It has the information, but the model’s ability to use it has degraded under the weight of everything else.

And it’s the same problem the MRCR v2 benchmark exposed: more context doesn’t mean better performance. Past a certain point, it actually gets worse. The information may still be there, but the model’s ability to find and reason over it erodes.

Here’s how virtual-context tackled the baseline step-by-step. First, it ingested the full 118,000-token haystack. It tagged every turn by topic, summarized by topic, extracted structured facts, and preserved session dates. After this phase, every detail from the original conversation was still stored and retrievable. Nothing was thrown away.

Then, instead of stuffing everything back into the context window, it gave the model tools to find what it needed:

The model could locate and retrieve any fact from any session on demand. This is what makes the smaller window possible. It’s not a limitation; it’s a choice. We could have matched the baseline’s 118K. But going smaller pulls two levers simultaneously. It reduces cost (fewer tokens processed per turn) and it reduces noise (less irrelevant content competing for the model’s attention). 55% fewer tokens and nearly three times the accuracy. 82% of questions were answered in just one or two tool calls. The model isn’t doing exhaustive searches. It formulates a targeted query, gets the result, and answers.

Two things make this work: better compression, and the ability to get full detail back when the model needs it.

Compression occurs by topic, not simply by age. The standard approach to compaction summarizes the oldest turns first: a chronological block of turns 40 through 50 that might contain meal planning, vacation research, and home renovation all jumbled together. The summarizer faces competing interests. If it preserves the caterer’s pricing, it might drop the contractor’s timeline. virtual-context compacts by topic instead. All the meal-planning turns get summarized together. All the vacation turns get summarized together. No competing interests, no triage.

And when even summaries aren’t precise enough, find_quote searches across all stored conversation text, not just topics whose names seem relevant. In one benchmark question, the delivery date for a remote shutter release was stored under basketball-tournament, a completely unrelated topic. The model’s full-text search found it anyway. The topic organization is a navigational aid, not a cage.

The benchmark questions are mostly factoid lookups: “what hotel did I book?” or “when was my 5K race?” To show how the system handles something more complex, I asked an open-ended question against the same LongMemEval conversation haystack: “Can you put together a complete itinerary for my Chicago trip based on everything we discussed? Restaurants, hotel, transportation, and activities.”

This isn’t a factoid lookup. It requires synthesizing information scattered across five different topic areas in the conversation. Here’s what the model did:

The model built, searched, and derived connections, expanding whatever information it needed, and synthesized across topics. That’s not simple retrieval. That’s a contextual response.

The standard LongMemEval haystack is about 118K tokens. While that’s a decent-sized conversation, one moderately complex website development project can fill up a context window in a matter of hours, with older information regularly being compacted and fidelity lost.

To test what happens at an even larger scale, I merged multiple LongMemEval haystacks together into single massive conversations of 660,000 and 926,000 tokens, representing 6 to 10 months of daily conversation. Then I tested four frontier models (Gemini 3 Pro, GPT-5.4, Claude Sonnet 4.5, and Claude Opus 4.6) on questions they had already answered correctly at the standard 118K size.

They failed.

At 660K tokens, GPT-5.4 was asked “how many babies were born to friends and family?” and answered 6 instead of 5. It included an adopted child that didn’t meet the “born” criterion. The other three frontier models got it right at this scale, but the cracks were already showing.

At 926K tokens, the cracks became a collapse. The question: “What is the order of the three trips I took in the past three months, from earliest to latest?” Four frontier models attempted it. GPT-5.4 found the right three trips but reversed their chronological order. Gemini picked the wrong set of trips entirely, substituting a Dubai itinerary for a day hike. Opus 4.6 (the most capable and expensive model available, at $9.26 per query) confused a merely planned trip with a completed one. Sonnet’s 1M beta returned a server error and couldn’t process the context at all.

Zero out of four. The correct information was present in every case. None of the models could use it.

The same question, with virtual-context managing the context: four out of four correct. Sonnet, GPT-5.4, Gemini, and Opus all got it right, at costs ranging from $0.15 to $0.76 per query. Opus with virtual-context costs $0.69 and gets the answer right. Opus without it costs $9.26 and gets it wrong.

Context rot is a progressive disease and it’s not specific to one model. Models that work fine at 118K tokens degrade at 660K and break down completely near a million. The information doesn’t disappear; the models just can’t use it anymore.

virtual-context is the vaccine to this scaling problem, because the window it manages doesn’t grow with the conversation. A massive conversation or agentic AI payload always gets a curated ~25K managed window.

There’s a moment somewhere around 200,000 tokens into a conversation, where the conventional wisdom about AI deployment quietly breaks down.

Up to that point, brute force is defensible. But in any serious enterprise context, that threshold gets crossed faster than you would think. Costs start to creep up, and latency increases while accuracy degrades. The instinct is to reach for a bigger, more expensive model. Part 3 is about why that instinct is wrong.

The data shows us something counterintuitive: a mid-tier model with structured memory doesn’t just compete with a flagship model. With structured memory and context management, it wins, on every dimension, at a fraction of the price.

virtual-context is open source: github.com/virtual-context/virtual-context
official website at: www.virtual-context.com

Part 3 is coming soon.