I forced a Mixture of Experts to specialize by topic. It got worse.

The folk model says a Mixture of Experts holds a code expert and a medicine expert. Trained from scratch on two consumer GPUs, the routing tells you more about whether a word is a comma than about what the document is about, and it froze in the first 2% of training. Three attempts to force topic routing all came out slightly worse.