What's this about?
Massively parallel simulation changes the data regime that off-policy reinforcement learning trains in, which breaks the stabilisers designed for data-limited replay. Across eight benchmark families the paper shows these stabilisers are data-regime-dependent — parameter normalisation helps in one regime and not the other. 134 upvotes.
What is "off-policy" RL?
On-policy learning means you can only learn from actions your current policy took; every update throws the old data away. Off-policy learning lets you reuse a stored buffer of past experience — far more sample-efficient, but unstable, because you are learning from decisions a different policy made. Decades of tricks exist to stabilise that. WarpSAC's point is those tricks assume data scarcity, and when a GPU simulator gives you effectively unlimited fresh experience, they stop earning their place.
Why does this matter to you?
- Directly relevant to Microduck's pipeline, which is exactly the massively-parallel-simulation regime.
- "Your stabiliser is regime-dependent" is a useful general lesson: techniques adopted under one constraint outlive the constraint.
- Eight benchmark families is unusually thorough for this kind of claim.