What's this about?
Because a public benchmark may be in the training data, OpenAI built "ExploitBench – Internal Port (June–August 2026)": 20 high-severity V8 vulnerabilities disclosed too recently to be contaminated. On these, Astra achieved much higher arbitrary-code-execution rates than GPT-5.6 Sol, using far fewer output tokens.
What is contamination, and why does V8 make a good test?
Contamination is when benchmark answers are in the training data, so the model recalls rather than solves. Using vulnerabilities disclosed after the training cutoff is the standard fix. V8 — Chrome's JavaScript engine — is a good target because it is enormous, performance-critical, written in C++, attacked constantly by well-resourced professionals, and hardened accordingly. If a model produces arbitrary code execution in V8, that is not a toy result.
Why does this matter to you?
- "Far fewer output tokens" is the sleeper metric: efficiency determines how many attempts an attacker can afford, and it is improving faster than raw capability.
- Building your own uncontaminated benchmark from recent disclosures is a technique any security team can copy.
- Twenty vulnerabilities is a small sample, and the post does not report a per-bug breakdown.