What's this about?
A Rust library that tokenizes text at gigabytes per second — roughly 1000x faster than common tokenizers. It picked up 1.3k GitHub stars in a day.
What exactly is "tokenization"?
Before an LLM sees your text, it's chopped into "tokens" — word fragments the model actually reads. It's like dicing vegetables before cooking: mandatory prep work. It's a tiny share of total compute during chat, but huge for data-pipeline jobs that tokenize trillions of words to prepare training datasets.
What's the community saying?
HN 389 points, 77 comments
- Top joke: making something 1000x faster that's 0.1% of runtime is "the most software-developer thing imaginable."
- Defenders: dataset preparation, search, and tokenize-only services are genuinely bottlenecked here.
- The provocative follow-up: how many other 1000x optimizations are lying around the inference stack?