Gigatoken: making tokenisation care about the hardware
A first-principles tour of the SIMD scanners, cache-aware layouts, huge pages and worker scheduling behind Gigatoken.
Peter Bhabra
Peter Bhabra's blog.
Latest posts
Gigatoken: A Thousand Times Faster?
Gigatoken claims a 1,000x tokenizer speedup. My production-shaped tests delivered 30x to 40x gains in the tokenizer core.
The swarm that designs itself
I rebuilt Moonshot's Kimi agent swarm and pointed it at a real codebase: ~53× fewer tokens and ~45× cheaper than one long-context agent.