What's this about?
The paper linearly combines representations of two text streams, observes a corresponding blend of next-token distributions and introduces guided decoding that separates two coherent continuations from one model pass.
What exactly is linear superposition?
If signal A and signal B are added, a linear system produces output A plus output B. Finding this inside a nonlinear transformer suggests some computations can share one pass without becoming an indistinguishable mixture.
Why does this matter?
author-reported preprint; 75 HF upvotes.
- The effect appears intrinsic to the architecture rather than learned during training.
- Pretraining weakens it, while lightweight tuning restores it.
- Practical speed or memory gains outside the paper's decoding setup remain to be demonstrated.