Mixture of Experts: the sparse model that lights up 93% of itself

A Mixture-of-Experts model is supposed to spend a fraction of itself on each token. I put CUDA events around every MoE layer of OLMoE-1B-7B on an RTX 4090 and recorded which of its 1,024 expert slots actually fire. At a batch of eight, 628 of them fire every single step -- and at batch 32 it is 93%. The sparsity is real, it is just not where people think it is.