Mojo-tokenizer: Fastest AI Token Output → Readable Text on Apple Silicon?

0Sign in to vote or save

This site does not allow itself to be embedded. You can still read it on the original site — the toolbar below keeps your place in the directory.

mojo-tokenizer decodes tokens at 144M tokens/sec on Apple Silicon — 3.1x faster than tiktoken and 1.2x faster than rs-bpe. We benchmarked…

Comments

Nothing yet. Say the first thing.