How LLMs Work, Part 1: Tokenization and Embeddings

When a new model is announced, we often see a description like this: Llama 2 7B: 4096 dimensions, 32 attention heads, 32 Transformer blocks What do those numbers mean? And when you send a prompt like this: The cat sat on the mat what actually happens under the hood? Those questions sent me down the rabbit hole of trying to understand the internals of large language models. This post is my attempt…