When over 3 million pages of the Epstein Files were released, you might have wondered, “Can I just dump everything into ChatGPT and ask for the biggest revelations?”
I hope you didn’t actually try it, as you would’ve wasted your time — because of something called the “context window” of an LLM.
There are only so many words an LLM can work on. The “context window” is the limit. The LLM doesn’t see anything that’s not in the context window.
Today’s LLMs’ context windows can take in 1 million tokens, which is about 750,000 words … which is about 1,500 pages.
Reminder: Epstein Files have over 3 million pages.
That would need roughly a 2-billion-token context window LLM. Which doesn’t exist today.
Maybe you’re thinking, “Pfft, forget about the Epstein Files, then. It’s an outlier case. I’m sure the usual legal cases are not that enormous.”
Oh, why, I’m glad you brought that up.
So, a typical large lawsuit has hundreds of thousands of pages involved; maybe even a million pages.
Let’s say a case has 500 thousand pages (evidence, transcripts, emails, contracts, filings, etc.). This would need a 330-million-token context window to fit in. And that’s not even counting the lawyer’s back and forth conversation with the LLM.
To put things into perspective:
So having an ever bigger context window is the holy grail, right?
It would be every law firm partner’s dream — to use an LLM and get quick results instead of paying a team of junior associates to spend months reading everything.
And of course, the big LLM players leaned into that.
Sundar Pichai (Google/Alphabet CEO) was perhaps the most high-profile cheerleader for having bigger and bigger context windows. Especially when he said this during the Google I/O 2024 event:
One million tokens is opening up entirely new possibilities. It’s exciting, but I think we can push ourselves even further. So today, we’re expanding the context window to 2 million tokens, and making it available for developers in private preview. It’s amazing to look back and see just how much progress we’ve made in a few months. And this represents the next step on our journey towards the ultimate goal of infinite context.
If only life was that simple.
Researchers found that LLMs’ performance starts to drop after about 16k tokens (which is only about 1% of a 1-million-token context window), where they start missing out details and getting confused. Some researchers call this phenomenon “context rot”.
Some of the more astute among you might be asking, “Why do the donkey work of cramming everything into the context window? Why not just get the LLM to search all the documents and choose only the important details to put in its context window?”
Yes, that’s what Retrieval-Augmented Generation (RAG) does.
Sure, all serious legal AI products today use RAG. They’re well aware that the 1-million-token windows aren’t enough for complex litigation, and stuffing a context window is expensive and slow.
Info box: What’s RAG
Retrieval-Augmented Generation (RAG) is an architecture that lets an LLM get info from outside its context window, like from the internet or your personal documents storage.
How it works
You ask the LLM something
↓
The retriever (a search program that’s not part of the LLM) searches the external info source
↓
The retriever puts the most relevant chunks of content into the LLM’s context window
↓
And just like that, the LLM can now see the most important info to do its job — without you dumping entire documents into its context window
But, again—if only life was that simple.
If there are 500,000 pages of documents, the retriever might only pass the top 50 chunks of contents (out of, say, a total of 1 million chunks of text) to the LLM.
And since the LLM never sees the remaining documents, its answer is limited by what the retriever selected.
The billion-dollar question is: did the retriever give the right 50 chunks of content?
Maybe not!
The thing is, in many legal cases, you’re not looking for a needle in a haystack. Often, the evidence isn’t a needle, but a pattern of hay in the haystack.
It’s more realistic that no single document contains the answer; the answer emerges from relationships across documents.
Example:
Email A: Engineer reports issue
Email B: Manager acknowledges issue
Email C: Executive says delay disclosure
Document D: Public statement says issue unknown
These two statements CANNOT both be true:
The executive privately said, “We know there’s a problem, but let’s delay disclosure.”
And later the company publicly said, “We are not aware of any problem.”
The contradiction only becomes apparent when you connect information across multiple documents.
A RAG system may retrieve only Document D because it mentions the public statement, or only Email A because it contains the technical issue. If the other document looks unremarkable in isolation, they may not rank highly enough to be retrieved. And so, it may miss the crucial cross-document pattern.
There are techniques to improve discovering cross-document relationships (e.g., perform iterative retrieval, build knowledge graphs, run multi-hop retrieval), and they do help.
But none can guarantee that the retriever won’t overlook a seemingly irrelevant document on its own but is actually needed to reveal the pattern.
So no, bigger context windows don’t replace a human carefully reviewing the evidence. And no, RAG doesn’t quietly fix it behind the scenes either. You just get a different flavor of “the machine drowned the important detail in a sea of context.”
Of course, humans miss things too.
But in high-stakes work like litigation, missing something isn’t the end of the process. Lawyers revisit the evidence, change their search strategy, test alternative hypotheses, and re-examine documents in light of new discoveries.
Bigger context windows and RAG can assist that process, but they don’t replace it.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.