What's this about?
Agents explore a running application and recreate it across Ubuntu, macOS, Windows, Android and web. The benchmark combines visual assertions with programmatic behavioural tests.
What exactly is reference-grounded evaluation?
The original application acts as an oracle for expected behaviour. This is more concrete than visual resemblance alone, but hidden tests still cover only selected interactions.
Why does this matter?
Author-reported preprint; 7 HF upvotes at fetch time.
- Real computer work interleaves interface navigation and coding.
- A 58.1% leader score shows substantial room for progress.
- Only 2.8% full programmatic passes exposes brittle partial success.