Towards a Bitter Lesson of Optimization: When Neural Networks Write Their Own Update Rules
We explore what can be the future of neural network parameter optimization
Sifal Klioui
A blog about Machine learning, programming, and other interesting things.
Latest posts
Why Modern LLMs Dropped Mean Centering (And Got Away With It)
Visualizing the hidden 3D geometry behind Layer Normalization and uncovering the mathematical trick that makes RMSNorm tick.
The Epsilon Trap: When Adam Stops Being Adam
Beyond numerical stability, we investigate an often overlooked hyperparameter in the Adam optimizer: epsilon.
Entropic Instruction Following: Does Semantic Coherence Help LLMs Follow Instructions?
Testing whether semantic relatedness of instructions affects a model's ability to follow them under cognitive load
Elements Of Mechanistic Interpretability: From Observation to Causation
We strip down mechanistic interpretability to three key experiments: watching a model 'think', finding where it stores concepts, and performing 'causal surgery' to change its 'thought process'