What's this about?
Training AI to interact well with humans usually involves having one large language model role-play the user, so the system can practice at scale. This paper shows that approach systematically fails to generalize, and traces the cause to what it calls simulator collapse — the single frozen simulator doesn't represent the diversity of actual people.
What exactly is a "simulator" in reinforcement learning?
Reinforcement learning needs enormous numbers of practice attempts, far more than real humans can provide. So researchers substitute a model that plays the human: it responds to the agent, gets frustrated, changes its mind. The agent then optimizes against that stand-in. The risk is obvious once stated — the agent becomes excellent at satisfying the simulator, and the simulator is one model with one set of habits. It's like training a customer-service rep exclusively against one actor who always plays the same customer.
Why is this worth noticing?
- Model-simulated users are now standard practice in agent training, so a systematic generalization failure affects a lot of current work.
- It connects to a broader worry about models training on model-generated data, where diversity quietly collapses in ways that only show up on contact with reality.