How LLMs Generate Text

This is Part 2. Part 1 covered attention, how tokens look at each other and share information. This post covers how that machinery generates text one token at a time, and why long contexts are expensive. I found the generation loop itself surprisingly simple — the complexity is in what it costs.