Optimizing training a GPT style Tokenizer with C++

Introduction In February 2024, renowned AI researcher and educator Andrej Karpathy, published the following video. https://www.youtube.com/watch?v=zduSFxRajkE An excellent introduction to the fascinating topic of tokenization, something he considers a necessary evil right now. Maybe the need for it will go away but for now it must be well understood by AI engineers in order to get good results…