Orthrus and lossless decoding

What's this about?

Orthrus studies speculative decoding under practical BF16 computation. In one test, only 45% of generated trajectories matched the baseline exactly; the authors also inspect downstream performance rather than assume numerical identity.

What exactly is speculative decoding?

A small or parallel process guesses upcoming tokens, and the main model verifies them to speed output. “Lossless” may depend on exact arithmetic and implementation, not just the algorithm on paper.

What's the community saying?

1. Runtime precision can break exact-output claims.
2. Small numerical differences may not always hurt task quality.
3. Both trajectory fidelity and real downstream metrics should be reported.