Build a (small) language model by counting

This week we continue our exploration of Large Language Models by building a simple markov based language model. By the end of this post you’ll understand better the basic mechanisms underlying language generation.

You can also find the companion notebook on the LLMs for Data Science GitHub repository:

You can also find the companion notebook on the LLMs for Data Science GitHub repository:

LLMs for Science GitHub

Every language model does one thing. It reads some words and guesses the next one. ChatGPT does it. Your phone keyboard does it. The model in this post does it too, and you can build this one by hand. It counts how often each three-word sequence is followed by a given word, then turns those counts into probabilities. That single idea, next-word prediction, sits at the center of every system in the field.

The build is short. We start with what an n-gram model is and where the idea came from. We load WikiText-103 and study it. We build the model, read off what it learned, and let it write. We close by set…