Bias Compounds, Variance Washes Out

This site does not allow itself to be embedded. You can still read it on the original site — the toolbar below keeps your place in the directory.

Round-to-nearest makes the same rounding error every time. Stochastic rounding makes a different error each time, centered on zero. When the same error repeats, it compounds. When errors are zero-mean, they partly cancel. Add 0.001 to 1.0 a thousand times in BF16 and round-to-nearest never moves. Every update falls closer to 1.0 than to the next representable value, so every update rounds back to…

Read on convergentthinking.sh ↗