What's this about?
Trail of Bits says a 26% “clean fix” headline leaned on invalid test setups and grader disagreement. Its reanalysis says 2,634 of 3,067 patches—86%—blocked supplied exploits under reasonable conditions, without claiming all underlying bugs were completely fixed.
What exactly is a patch benchmark?
It gives an agent buggy code and tests whether its edit works. Passing a known attack test is narrower than proving the program secure and regression-free; independent review is still needed.
What's the community saying?
HN 36 points, 9 comments.
- Rechecking test conditions and human baselines is essential.
- Exploit-blocking must not be confused with complete repair.
- Two model graders disagreed on many labels, making validation loops important.