Personal site and blog
Most of the time at inference, the Fisher pullback through the unembedding tells you exactly how far you can perturb the residual stream before the predictive distribution moves — and it's literally the second-order Taylor expansion of KL.
A v1.0 rewrite of my old curvature analysis library: new operators, new algorithms, and a lot of numerical validation I should have written the first time.
Understanding how attention can express logical propositions about correlations between tokens and other fun tips and tricks
Understanding how attention can express logical propositions about correlations between tokens and other fun tips and tricks
Studying the covariance structure of mini-batch noise in stochastic gradient descent
MAML Hunting
An application of dynamical systems theory to optimization