What's this about?
Across 300 tasks and 3,000 valid/invalid candidate implementations, the joint success measure reaches only 28%, versus 59.67% on single-reference success. Tests must accept every valid implementation and reject invalid ones; TestHelix improves the joint measure by 8.67–9 points.
What exactly is reference implementation?
One known-correct way to solve a programming task. Other correct solutions may look different, so tests should check behavior rather than copy its internals.
What's the community saying / why does it matter?
HF: 5 upvotes; no substantive discussion collected. Author-reported results suggest testing evaluations need diverse valid and broken implementations, not just agreement with one answer.