What's this about?
A multi-difficulty benchmark for evaluating LLM judges on agentic tool-calling — measuring how well a model can assess whether another model used tools correctly. 17 upvotes.
Why does judging tool calls need its own benchmark?
Because it is harder than judging text and it is load-bearing everywhere. To know whether a tool call was right you need to know whether it was the right tool, whether the arguments were correct, whether it was necessary at all, and whether the result was interpreted properly — four separate judgements, most of which require understanding the task's state at that moment. And this capability sits underneath everything: automated evaluation, RL reward signals, and runtime safety monitors all assume a model can tell a good trajectory from a bad one. If judges are unreliable, every one of those systems inherits the unreliability silently.
Why does this matter to you?
- "Multi-difficulty" is the right design — a judge that handles obvious cases and fails on subtle ones looks fine on an averaged score.
- It is the measurement layer under the rubric-induction and critique-supervision papers from the last few days.
- Directly relevant to anyone using a model to triage another model's security findings.