The Technical User's Introduction to LLM Tokenization

This site does not allow itself to be embedded. You can still read it on the original site — the toolbar below keeps your place in the directory.

This blog post is basically my notes from Andrej Karpathy’s excellent YouTube talk Let’s build the GPT Tokenizer Table of Contents Introduction Why Does Tokenization Matter? Tokenization Details Naive Vocabulary Construction Example: Shakespeare Understanding Tokenization Challenges Interlude - Why Can’t we Just Use the Unicode Codes Points? Byte Pair Encoding (BPE) Algorithm What’s the Difference…

Read on christophergs.com ↗