Token Balancing

Mixture of Experts (MoEs) are very popular. They've been popular for a long time, but I'm currently catching up, so there's a lot of new things here for me. I think we generally all understand why imbalanced experts are bad; both from a learning perspective (experts don't improve if they don't get enough gradients) and a performance perspective (unbalanced routing means many GPUs idle). Recently I…