This site does not allow itself to be embedded. You can still read it on the original site — the toolbar below keeps your place in the directory.
“Attention normalizes the wrong norm” showed that attention output variance collapses as context grows. L2-normalized softmax fixes it in theory, but existing architectures are tuned for L1. So, swapping the norm breaks hyperparameters. There’s a simpler fix: normalize the output. av = attn @ v # or F.scaled_dot_product_attention(...) av = GroupNorm(num_groups = heads,…
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.