What's this about?
Request-level routing logic feeds asynchronous per-model subservers with delayed batching. Authors report 2.01–64.15× higher decoding throughput against tested systems across different routing workloads; this range is not a speedup guarantee for every ordinary LLM request.
What exactly is token-level routing?
Choosing which model produces each small piece of an answer, rather than choosing one model for the entire request.
What's the community saying / why does it matter?
HF: 28 upvotes; no substantive discussion collected. Useful for specialized serving infrastructure; independent replication and comparison details matter before extrapolating the speedup.