In the age of LLMs, token usage is a recurring topic

Note: I wrote this blog post for work . Just cross posting here for my own records. When we first started building RAG systems with early models, context windows were tiny (like 8k) and it took some engineering cleverness just to figure out how to process single files effectively. Now context windows are much, much larger, in the hundreds of thousands to millions of tokens range, but our systems…