When Equivalent Weights Train Differently
Why coordinate-level optimizers can behave differently on weights that represent the same model, and how quotient-aware updates remove the hidden gauge.
Ji-Ha's Blog
Ji-Ha Kim's blog for sharing things.
Latest posts
Fast Tight Spectral-Norm Bounds
A GPU-friendly algorithm for tight certified singular-value endpoint bounds using scaled Gram matrices, SYRKs, Frobenius reductions, and a small scalar moment problem.
Autoregression vs Diffusion - Understanding Sampling via Optimal Transport
Autoregression and diffusion look like opposites, but both are solving the same transport problem - how to turn simple noise into structured data.
Fast Polar Decomposition for Muon Optimizer with Rational and Polynomial Iterations
A hardware-aware hybrid polar decomposition for ML: one Dynamic Weighted Halley (rational) step to handle the hard early regime, then two Polar Express (polynomial) cleanup steps once the spectrum is easy. The result is exactly two rectangular GEMMs, no eigendecomposition or power iteration, and robust convergence from condition numbers up to 1000.
Lion-K CCWD: Corrected Cautious Weight Decay
Retention and radius parametrization for Lion-K with Corrected Cautious Weight Decay.