Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning

It takes many people to make The Luddite happen. Many of these people also do cool things outside The Luddite . With all the AI hype, we are crossposting one of those things — a paper on tokenization in Large Language Models. Hopefully, this will be a nice counterbalance to all the bad AI research that I've covered. Abstract Tokenization is a necessary component within the current architecture of…