This site does not allow itself to be embedded. You can still read it on the original site — the toolbar below keeps your place in the directory.
We treat attention as a solved primitive. You can fuse kernels, tile memory access, quantize weights, but the formula itself is finished. It isn’t. Softmax normalizes the L1 norm to 1. Variance preservation requires the L2 norm to equal 1. These constraints differ. The mismatch causes attention output variance to collapse as sequence length grows, forcing models to learn position-dependent…
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.