Muon Optimizer Explained: Newton-Schulz Orthogonalization Beyond Adam

Muon stands for M oment U m O rthogonalized by N ewton-Schulz and was invented by Keller Jordan . The key idea: Instead of applying Adam-style per-element adaptive updates to model parameters, Muon orthogonalizes the momentum matrix before using it as the update direction. Table of Contents Adam Optimizer Matrix Orthogonalization Newton-Schulz 5 Iteration Muon QK-Clip Multihead Latent Attention…