LLMs Part 5: Tokenisation

This is Part 5 in my series on building language models. For more, see Part 1 , Part 2 , Part 3 and Part 4 . In this post, I explore 3 different tokenisation approaches and show you the outcome of each as they are subtly different. Introduction You’ll often hear it repeated in tech discussions that LLMs “are just next-token prediction machines”. Having now built a few, I have to reiterate how very…