What's this about?
LLMs perform remarkable multi-hop reasoning over long contexts — connecting a fact on page two to one on page ninety — yet the internal mechanisms enabling those distant leaps are poorly understood. This paper measures the topological compression that happens in long-context representations.
What exactly is multi-hop reasoning, and why is distance hard?
Multi-hop means chaining facts: A relates to B, B relates to C, therefore A relates to C. When A and C are far apart in a long document, the model must somehow bring them together internally. Attention in principle connects any two positions, but the mechanism is not free — with hundreds of thousands of tokens, most pairs cannot be meaningfully attended to. The "six degrees" framing suggests the model builds something like a compressed map where distant points become reachable in a few hops, similar to how social networks connect any two people through a short chain despite their size.
Why is this worth noticing?
- It's a mechanistic account of something usually measured only behaviourally — we know models do this, and largely don't know how.
- It has practical implications for the context debates running all month: if long-context reasoning depends on a compression structure, that suggests why quality degrades unevenly with length rather than smoothly.