Mixture of Experts (MoE): How Transformers Scale Without Activating Everything.

Mixture of Experts (MoE) is one of the main techniques used to scale modern language models without making every token pay the full computational cost of the model. The basic idea is surprisingly simple: instead of sending every token through one enormous feed-forward network, we split it into many smaller expert networks and only activate a few experts for each token. These notes walk through the…