3.3x Faster HuggingFace Tokenizers for Single Sequence

I made HuggingFace tokenizers 3.3x faster by parallelizing single-input tokenization with overlapping chunks, zero-copy offset operations, SIMD-accelerated boundary detection, and cache-hierarchy-aware chunking. The result is bit-identical to serial encoding. Fast tokenization is critical to achieve low TTFT given long context. The Long Context Revolution Has a Bottleneck LLM models now routinely…