What's this about?
Humans learn continuously from experience, while conventional LLM evaluations ignore whether a model can improve through inference-time interaction at all. This paper studies that capability directly, proposing Chain-of-Experience for continual improvement.
What exactly is "improving at inference time"?
Normally a model's abilities are fixed when training ends — using it doesn't make it better. Improving at inference time means the system gets better from its own accumulated attempts without any weight updates: it records what worked, what failed and why, and consults that record on future attempts. The distinction from memory is that memory stores facts, while this stores *procedural* lessons — not "the API key is X" but "the last three times I tried this approach it failed for this reason."
Why is this worth noticing?
- It's the same target Rich Sutton's new lab is aimed at, from the opposite direction — Oak Lab is building agents that keep learning, this measures whether current ones can.
- It's also what StateM's runbooks implement in practice: postmortem findings becoming persistent executable preconditions is Chain-of-Experience with an engineering answer.
- Evaluating this at all requires a different setup, since a benchmark that runs each task once cannot detect improvement across attempts.