This site does not allow itself to be embedded. You can still read it on the original site — the toolbar below keeps your place in the directory.
This was a fun project I worked on in November 2023. Tokenization is an oft-neglected topic in natural language processing. With the recent blow-up of interest in language models, I thought it might be good to step back and really get into the guts of what tokenization is. I created a repository to serve as a deep dive into different aspects of tokenization. This was initially supposed to be a…
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.