What's this about?
A benchmark mines 1,975 replay-verified behaviours across 26 apps and builds 4,063 tasks; the authors report 49.2% vs 28.8% full-app workflow success for Astra and Opus 5 (11 HF upvotes).
What exactly is replay-verified behaviour?
A sequence of user actions that can be executed again and checked against a working reference app. It tests functioning behaviour rather than whether code merely looks plausible.
What's the community saying?
11 HF upvotes; author-reported preprint, no major discussion sampled.
- Significance: reference apps supply executable acceptance targets.
- Significance: deeper reconstruction tasks expose cumulative failures.
- Significance: independent replication and app selection still matter.