The Attention Mechanism: Why Your LLM Forgets the Middle

This site does not allow itself to be embedded. You can still read it on the original site — the toolbar below keeps your place in the directory.

LLM context rot isn't a bug. It's a mathematical property of how the attention mechanism works. Three interacting failure modes (softmax's sum-to-one constraint, RoPE's distance decay, and causal masking's primacy bias) combine to produce the U-shaped attention curve where models attend well to the beginning and end of context but lose the middle. Understanding the mechanism changes how you…

Read on duncanleung.com ↗