What's this about?
Annotators identify the first wrong token, replace it and let the model continue, producing both a corrected response and precise supervision about the error.
What exactly is on-policy data?
It stays close to what the current model itself would generate. That reduces mismatch between training examples and the model's real error distribution.
Why does this matter?
Author-reported preprint; 19 HF upvotes at fetch time.
- Fine-grained corrections can teach where a trajectory first went wrong.
- The 52% time reduction comes from a small controlled study.
- Tool-connected annotation may extend beyond plain text responses.