stuff
When I started writing the flash attention series, embeddings were on my “I get it intuitively” list. A word becomes an N-dimensional vector, the geometry encodes meaning, cats are near dogs and not near typewriters. That was enough to write the forward pass. For GPU code, embeddings are floats with shape and the kernel doesn’t care what the floats mean.
This is the fifth post in a series on LLM internals. Part 1 covered attention, Part 2 covered generation, Part 3 covered the Flash Attention algorithm, Part 4 put it on a GPU with Triton. This post takes the Triton kernel from Part 4 and ports it to a TPU.
This is the fourth post in a series on LLM internals. Part 1 covered attention, Part 2 covered generation, Part 3 covered the Flash Attention algorithm. This post takes Part 3’s numpy simulation and puts it on a GPU where the SRAM/HBM distinction is real.
This is the third post in a series on LLM internals. Part 1 covered attention. Part 2 covered generation. This post: the Flash Attention algorithm and why it matters for long contexts.
This is Part 2. Part 1 covered attention, how tokens look at each other and share information. This post covers how that machinery generates text one token at a time, and why long contexts are expensive. I found the generation loop itself surprisingly simple — the complexity is in what it costs.
I gave a talk to my team about how LLMs work under the hood. This is that talk turned into a blog post. There are three parts planned. This first one is about attention.
On the off chance that you stumble upon this, I feel sorry, but in case you are wondering, this is not an apology.
The fantasy of creating something from absolute nothingness, and to build it into existence with my bare hands, is a very appealing proposition. Truth be told, I am addicted to that fantasy. But when I dream of a fantasy, I tend to ignore a lot of intricate details, and that would be the journey of getting there.
Sometimes, it’s almost shocking how fast a year goes by.