A Few Gaussians Is All You Need:
SSOG-Attention That Steers Instead of Scores
I swapped the transformer's matchmaking service for a handful of Gaussians, and it beat content scoring without comparing a single pair of tokens.
Crack open any vision transformer and you'll find the same machinery humming at its core: scaled dot-product attention (SDPA). Every token asks every other token "how much do I care about you?", and the answer is a big $N \times N$ matrix of similarity scores, computed from the content of the tokens themselves.
We rarely second-guess this. I know, because I tried once before: a while ago I swapped the dot product for an RBF kernel and learned a lot about what similarity even means in these systems. This time I wanted to question something more fundamental:
Why does the network need to compute "where to look" from scratch, for every image, from content?
Think about how you read an image. When you process a patch on a bird's wing, you don't run a similarity search against all other patches. You already know where to look: a bit left, a bit up (is that the head?), further out (where does the wing end?). Where to look is mostly a function of geometry, not content. Content only fine-tunes it.
So I built attention that way. Each head owns a few Gaussians, a small handful of numbers in total, forming a fixed field over relative position. A learned habit of where to look. Then the trick that makes it actually work: a tiny content-conditioned nudge that lets each token shift its field. No query-key dot products anywhere. Content never scores, it only steers.
The results surprised me. The fixed field alone, completely content-blind, comes within one point of SDPA on ImageNet. Steering closes that gap entirely, and the full model beats the baseline. On small data the geometric prior is worth a ridiculous +17 points. It scales too: a 12M version reaches 72% on ImageNet, ahead of its SDPA twin while being 20% smaller and 30% cheaper to run. The kicker: since the field factorizes into two 1D filter passes, the $N \times N$ attention matrix never exists, so this is attention that scales near-linearly instead of quadratically. Best of all, "what did attention learn?" stops being a heatmap-and-a-shrug question. Every head is just a few blobs you can literally plot and read with a ruler.
