What's this about?
Combining language models with reinforcement learning is increasingly common, but the theoretical status of LLM-derived reward signals is usually left implicit. This paper formalizes the hybrid setup and provides a framework where LLM feedback shapes rewards without changing which policy is optimal.
What exactly is "policy-invariant" shaping?
Reward shaping means adding extra reward signal to guide learning — rewarding progress toward a goal rather than only the goal itself, so the agent isn't learning from nothing. The danger is that badly-chosen shaping changes what's optimal: reward an agent for approaching the target and it may learn to circle it forever rather than arrive. There's a classical result showing that shaping of a particular mathematical form provably cannot change the optimal policy, only how fast you find it. Applying that guarantee to LLM-generated feedback is the contribution — you get the model's judgment as guidance without it silently redefining the objective.
Why is this worth noticing?
- LLM-as-judge is now standard practice for domains where rewards are hard to specify, and it's usually done without any guarantee that the judge isn't distorting the goal.
- It's the formal counterpart to Trajectory Labs' argument this week that post-training algorithms need redesigning for non-verifiable, per-token rewards — same problem, one taking the theory route.