Recent content on Convergent Thinking
The phrase means blank slate, but the object was stone. A tablet, once carved, never goes back. You can fill the marks and cut new ones, but the uncut surface is gone. While newborns hear every language, by the first birthday, a child will only hear its own. Foreign sounds may still arrive, but as familiar ones, with an ear shaped by what it heard most. No one shows a four-year-old a slasher film.…
Some problems get easier as you go, and others can’t start until they’re already solved. A sentence is the first kind. Each word tightens what can follow, and the ending arrives as if it were always going to. Sudoku is the second. Place a number and it forecloses options in cells you haven’t looked at; erase it and everything built on it collapses. The grid resolves only once you…
What separates training methods that scale from those that don’t? Across 61,000 configurations on CIFAR, batch normalization, orthogonalized updates, and ASAM help at every data scale with no sign of diminishing returns. Dropout and random weight perturbation are neutral to mildly helpful only at low strength, and the threshold drops as task complexity rises. BN orthogonalizes…
Round-to-nearest makes the same rounding error every time. Stochastic rounding makes a different error each time, centered on zero. When the same error repeats, it compounds. When errors are zero-mean, they partly cancel. Add 0.001 to 1.0 a thousand times in BF16 and round-to-nearest never moves. Every update falls closer to 1.0 than to the next representable value, so every update rounds back to…
People optimize for local objectives and lose sight of global ones. The shape is always the same: you win at the thing directly in front of you, and in doing so, you lose something larger. Most tasks are very local, and in ML this has become extreme. We train on language and evaluate on language. It’s the same domain, so it must be fair. But “all human knowledge” turned out to be…
“Attention normalizes the wrong norm” showed that attention output variance collapses as context grows. L2-normalized softmax fixes it in theory, but existing architectures are tuned for L1. So, swapping the norm breaks hyperparameters. There’s a simpler fix: normalize the output. av = attn @ v # or F.scaled_dot_product_attention(...) av = GroupNorm(num_groups = heads,…
The hardest box to escape is the one you cannot see. For years, the entire field optimized next-token prediction. Lower perplexity meant a better model. This was the metric, the target, and the definition of progress. Then o1 trained for reasoning instead of prediction, and it turned out to be better at the things we actually cared about. The box was only visible once someone stepped outside.…
We treat attention as a solved primitive. You can fuse kernels, tile memory access, quantize weights, but the formula itself is finished. It isn’t. Softmax normalizes the L1 norm to 1. Variance preservation requires the L2 norm to equal 1. These constraints differ. The mismatch causes attention output variance to collapse as sequence length grows, forcing models to learn position-dependent…
When people say “X is the new oil,” they mean the bottleneck. The resource everyone needs and no one has enough of. For AI, that used to be compute. It isn’t anymore. Era of Oil During the industrial era, oil was the constraint, but people were abundant. You could always find another worker, but you couldn’t always find another barrel. Oil was what you fought wars over.…
I’m Lucas Nestler. I adjust numbers until other numbers get smaller. Find me elsewhere: Twitter · GitHub · nestler.sh