What's this about?
An experiment gave agents AI-research reproduction tasks and found repeated failures and human interventions, suggesting today's systems do not yet close a reliable research loop.
What exactly is research reproduction?
It means rebuilding a paper's method and verifying its result from the description, rather than merely summarizing it—a test of coding, judgment, debugging and self-checking together.
What's the community saying?
HN 66 points, 67 comments
- Skeptics of fast takeoff saw poor self-checking as a central bottleneck.
- Critics called Opus 4.8, OpenClaw and the roughly $3k budget stale compared with current frontier runs.
- Commenters distinguished models improving a future model from one model updating its own weights online.