Everyone talks about KV cache, almost nobody structures prompts so it actually matters.
Then they stare at the metrics, see “KV cached” in the logs, and the bill is the same, latency is the same, and nothing feels different.
In a transformer, every token gets a key and a value representation in every layer. The model reuses these when it generates the next tokens.
Normally, for every new token, you’d have to recompute attention over the entire previous context. That’s the expensive part. KV cache just means you keep the already computed key/value tensors, and for new tokens you only compute the new slice, then concatenate it with the cached part.
That’s it. KV cache is not magic. It just avoids recomputing the same prefix.
If the prefix is stable, you win big.
If the prefix keeps changing, you win nothing.
On paper, KV cache is beautiful. In production, it dies where prompt structure is treated as an afterthought.
Most applications have a three-layer prompt:
relatively stable system prompt / role / style / rules
some “container” layer: tools, schemas, format explanations
dynamic user input, context, fresh data
The story goes wrong when the dynamic part creeps up to the front, or starts to get mixed into what you thought was “static”.
A few very common cache killers:
Request-specific “system status”, timestamps, request IDs dumped at the top on every call.
Tool definitions that show up in a new order, with new IDs, on each request.
Log-like blocks glued to the front of the prompt because “it’s convenient to see it there”.
With this, the part that could have been identical in 99 out of 100 calls is now token-level different in every single one.
From the model’s perspective, there is no such thing as “almost the same”. It’s either the exact same token sequence, or it isn’t. If it isn’t, your cache advantage is gone.
Transformers generate left to right. Each new token attends to all previous tokens. KV cache follows that logic: you can reuse the key/value set for the prefix that hasn’t changed.
If the dynamic part is at the end, life is simple:
Stable prefix: cached.
Only the fresh tail needs new KV.
If the dynamic part is at the beginning, every call has a different prefix. Token positions shift. Key/value tensors are different. The model basically sees a completely new sequence.
On paper you “enabled KV cache”. In reality you have no reusable prefix.
This is not a gentle, gradual loss. It’s binary. Either the prefix is the same, or it isn’t.
Doesn’t matter.
Because of positional encoding, when you shove a few new tokens into the front, the positions of all later tokens change.
The model no longer sees:
[A, B, C, D, E]
It sees:
[X, A, B, C, D, E]
That [A, B, C, D, E] is not the old [A, B, C, D, E] anymore. Every token is tied to a different position. Your cached KV tensors were computed for the old positions, not the new ones.
So if you prepend 10 dynamic tokens (time, user meta, whatever) to every request, your entire system prompt and context gets recomputed for every single call.
While you sit there thinking “we’re fine, we have KV cache”.
There’s no magic trick here. Just discipline.
You want the most stable, least frequently changing parts at the front, and you want them stable not just conceptually, but token-wise.
A sane order looks something like this:
System prompt / role / style / global rules.
Tool definitions, schemas, format descriptions that rarely change.
Optional, semi-static layers (app-level config, instruction templates), kept as stable as possible.
At the very end: dynamic context + user query.
The key point: don’t push dynamic noise into positions 1–2.
If you really insist on having metadata, timestamps, request IDs, logs, put them at the end. Or into a separate channel if your infra supports that. For normal QA, the model almost never actually needs this junk.
If you’re hacking together a small demo, you probably don’t.
Once you’re in production and you have calls coming in every second, it starts to matter a lot whether you recompute 80% of your context every single time.
Say your stable part is ~6k tokens, dynamic part is ~1k.
With a KV-friendly structure, you compute the 6k once (it lands in cache), then only pay for the 1k tail each call.
With the “dynamic at the top” structure, you recompute the full 7k for every request.
You don’t need a PhD, or even a calculator, to see that the second version is multiple times more expensive.
And that’s just one model call. Add tool-calling, multi-turn dialogs, RAG, retries, regenerations – the waste stacks up.
KV cache only saves you money if you give it a real, fixed prefix. Without that, all you have is a toggled feature flag.
It’s not just what you put where. It’s how stable it is at the token level.
A few surprisingly expensive habits:
Random IDs at the top of the prompt. Tool IDs, session IDs, whatever. Don’t put them there.
Ever-changing numbering, dynamically generated bullet lists inside the “stable” area.
Frequently rewritten system prompts for the same role (“You are a helpful assistant”, “You are a senior AI engineer assistant”, and so on). If it’s a system prompt, freeze it.
Autogenerated tool definitions where formatting, ordering, wording changes with every build or app version.
KV cache doesn’t see “structure” or “concepts”. It sees a sequence of token IDs. Any noise in that sequence is enough to turn “almost the same” into “can’t reuse this at all”.
This is the point where “prompt engineering” stops being about wordsmithing and becomes a plain infra question.
If you’re running tens or hundreds of thousands of calls a day, you need to:
define what stable prefix means in your system,
keep it in a separate layer, from a separate source,
and block any dynamic junk from leaking into it.
This shouldn’t be something you drag around in a prompt builder UI. It should be structured like any other request pipeline.
The “text” is an API contract at this point. If you keep poking random holes into it, you’re just eating your own cache advantage.
Push all dynamic, user- or request-specific information to the bottom of the prompt.
Timestamps? Bottom.
User metadata? Bottom.
Chat history? After the system prompt, tools, and rules. Always.
Yes, the model still sees it. Yes, it can still attend to it. No, it doesn’t become “more important” if you shove it to the front. It only becomes more expensive.
Once you do this, KV cache stops being a marketing bullet and starts acting like an actual speedup.
You can get fancy later. But as long as your dynamic mess sits at the top of the prompt, there’s not much to talk about.
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.