How Attention Became Efficient & Scalable: KV Caching, MQA, GQA, MLA, and Sparse Attention.

Attention mechanisms have evolved considerably to make transformer inference faster and more memory-efficient. These notes trace that evolution: from vanilla self-attention , through KV caching , to memory-saving variants like MQA , GQA , and MLA , and finally to sparse attention methods like SWA and Deepseek Sparse Attention (DSA) . Table of Contents Self-Attention Masked (Causal) Self-Attention…