On Getting Lost In the Context and Positional Bias
Language models often struggle to retrieve information from the middle of their context. We study similar phenomena in a toy transformer and discuss how to fix it.
Bling Theory
Stories of things from theoretical physics.
Live Last read · last published · next check
Latest posts
On Catastrophic Forgetting
My experiments with neural networks that forget things in alarming ways.
A (Gentle) Introduction to String Theory: The Strings Themselves
Part 2 of the gentlest possible introduction to string theory, where I explain the strings themselves and what they do for quantum gravity.
A (Gentle) Introduction to String Theory: Particles and Gravity
What is string theory and what does it have to do with the rest of physics? Part 1 of a 2-part post, with some background on particle physics and gravity.
Bling theory
physics and mystery and fun