What's this about?
Models are increasingly trained and served with very long contexts spanning whole documents, code repositories and interaction histories, on the implicit assumption that longer is better. This paper finds a cost: long-context training undermines the model's parametric knowledge — what it knows from its weights alone.
What exactly is "parametric knowledge"?
There are two ways a model can know something. It can have absorbed it during training so it's stored in the weights — that's parametric knowledge, what it recalls without being told. Or it can read it from the text you supply right now, which is contextual knowledge. Think of an experienced doctor's internalized medical knowledge versus what they read in your chart. The finding is that training a model to lean harder on the chart appears to weaken what it carries in its head.
Why is this worth noticing?
- The whole industry is racing on context length as a headline number, and this suggests a trade-off that headline doesn't reveal.
- It has direct architectural implications for retrieval-augmented systems: if long-context training erodes recall from weights, systems that always retrieve may end up depending on retrieval being right.