Cache Management in Large Language Models

KV cache optimization is crucial for LLM deployment. It reduces latency by avoiding recomputation of past tokens, lowers operational costs with lower compute requirements, and enables deployment on resource-constrained devices by managing large memory footprints and optimizing data transfers. Recent advancements in LLMs have led to a rapid, industry-wide increase in supported context window sizes. In recent years, modern models across different vendors have expanded from tens of thousands of tokens to hundreds of thousands, and in some cases, millions. This reflects a broader trend rather than the evolution of any single product line, underscoring the growing importance of efficient KV-cache management in large context inference.

Here is the repository: link

Guide preview link: Preview

Premium guide link: Guide

Naive servers allocate a large, continuous buffer for each request. But most real requests are short, so much of that memory goes unused. When memory is allocated and freed in different sizes, it causes fragmentation.

PagedAttention uses fixed blocks of 16 tokens and keeps a block table for each request. This reduces wasted memory to just a few percent, since only the last block of each request is partly empty. There is no external fragmentation, because any free block can be used for any request. The saved memory can be used to increase the batch size, boosting throughput.

The flat cache uses a chained hash to find block-aligned prefixes. The radix tree, on the other hand, stores every shared prefix across all cached sequences and splits at the exact token where they differ. This approach is useful when prefixes branch, like in multi-turn chats, agent loops, or beam search candidates. It is less helpful when most traffic uses the same system prompt with only a unique ending.x.

In a shared batch, a single large prefill can slow down the whole iteration. This causes all decoding streams to stutter.

With disaggregation, prefill runs on nodes optimized for compute. The resulting KV is sent over RDMA or GPUDirect to nodes optimized for bandwidth, which handle decoding. Each side can batch work based on its own bottleneck. There is a cost to moving KV per request, so this method works best with fast networks and high-volume traffic, but may not help with slow connections or short chat requests.

1. Read vLLM’s block manager while keeping Chapter 6 open for reference. The code and the concepts closely match.

2. Test your deployment using real prompt lengths, output lengths, and concurrency levels. Before optimizing further, see how close decoding gets to the HBM bandwidth limit.

3. Enable FP8 or INT8 KV cache in SGLang or TensorRT-LLM and measure the increase in throughput. Then check quality on long-context tasks rather than just looking at perplexity.