Gigatoken: making tokenisation care about the hardware

A first-principles tour of the SIMD scanners, cache-aware layouts, huge pages and worker scheduling behind Gigatoken.