What's this about?
AI agents are improving fast at cybersecurity tasks — when they can read the source code. This benchmark targets the harder and more consequential case: malware, firmware, and proprietary applications, where only compiled binaries exist. It's also designed to be contamination-free.
What exactly is "benchmark contamination"?
A benchmark only measures capability if the model hasn't already seen the answers. Since models are trained on enormous web scrapes, popular public benchmarks frequently end up in the training data, and the model recalls rather than solves them — scores go up while real ability doesn't. A contamination-free benchmark uses material that provably wasn't in training, often by being newly created. For security work this is doubly important, since write-ups of well-known malware are all over the internet.
Why is this worth noticing?
- Reverse engineering compiled binaries without source is where the genuinely consequential security work happens, and it's precisely the setting current agent evaluations skip.
- It's directly relevant to the day's spoofed-ClaudeBot campaign: defenders analyzing novel attack tooling do this work under time pressure, and it's a natural place to want agent assistance.