Language Modeling Without Neural Networks

Generating Shakespeare has become the &ldquo;Hello World&rdquo; of language models. 1 Recently, I&rsquo;ve been messing with alternative language models and came across unbounded n-gram models. These models are purely statistical and don&rsquo;t require optimizing weights or training. A year ago, I read the paper Infini-gram , which scaled an unbounded n-gram model to trillions of tokens. While…