What's this about?
Long-horizon AI agents mostly interact with users through sparse, selective text updates, leaving people unsure what the agent is doing or when to step in. JarvisBench tests a proposed fix: an always-on, Jarvis-style spoken mediator layer that sits between the user and a working agent, answering questions in real time without interrupting the worker, proactively reporting progress, and injecting user guidance back into execution. It was tested on 34 text-only tasks in the OpenClaw environment with GPT-5.5, Claude Opus 4.7, and Gemini/GPT-based worker agents.
What exactly is a "mediator" agent?
Rather than one agent doing both the work and the talking, a mediator is a separate layer whose only job is communication — like a project manager who checks in on an engineer's progress and translates it for a client, instead of the engineer stopping work to explain themselves every few minutes. The benchmark measures this on two axes: does mediation actually help the worker agent finish tasks better (when it relays useful guidance), and does it make the process more understandable to the human.
Why does it matter?
Preliminary results show mediation can improve task performance when guidance is injected at the right moments, but effectiveness depends heavily on which LLM powers the mediator — meaning this "missing middle layer" between users and agents isn't a solved problem yet, just a promising direction with a new yardstick to measure it by.