Abstract:We develop a mathematical framework that interprets Transformer attention as an interacting particle system and studies its continuum (mean-field) limits. By idealizing attention on the sphere, we connect Transformer dynamics to Wasserstein gradient flows, synchronization models (Kuramoto), and mean-shift clustering. Central to our results is a global clustering phenomenon whereby tokens cluster asymptotically after long metastable states where they are arranged into multiple clusters. We further analyze a tractable equiangular reduction to obtain exact clustering rates, show how commonly used normalization schemes alter contraction speeds, and identify a phase transition for long-context attention. The results highlight both the mechanisms that drive representation collapse and the regimes that preserve expressive, multi-cluster structure in deep attention architectures.
| Comments: | to appear as Proceedings of the ICM2026, Philadelphia, USA |
| Subjects: | Machine Learning (cs.LG); Mathematical Physics (math-ph); Dynamical Systems (math.DS); Probability (math.PR) |
| Cite as: | arXiv:2512.01868 [cs.LG] |
| (or arXiv:2512.01868v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2512.01868 arXiv-issued DOI via DataCite |
Submission history
From: Philippe Rigollet [view email]
[v1]
Mon, 1 Dec 2025 16:51:00 UTC (539 KB)
[v2]
Tue, 9 Dec 2025 14:40:27 UTC (539 KB)
[v3]
Wed, 7 Jan 2026 15:27:09 UTC (539 KB)
[v4]
Fri, 30 Jan 2026 18:12:44 UTC (539 KB)