What's this about?
The methodological point AISLE is right to press. Stenberg published the frontier systems' zero-result on August 24, publicly and timestamped, before AISLE ran anything. So this is not a vendor choosing a favourable comparison after the fact — the number they were measured against was already on the record and set by someone else.
Why is that rare enough to be worth a paragraph?
Because almost every AI capability comparison is retrospective and self-selected. A vendor runs both systems, picks the tasks, and publishes. Here the baseline was established by a third party with no stake, in public, in advance; the target was production code rather than a benchmark with answers that might be in training data; and the adjudication was done by curl's maintainers, who decided both whether each finding was real and whether it deserved a CVE. Those three properties together are close to the strongest external validation available for a zero-day discovery claim.
Why does this matter to you?
- It is a template for how to evaluate a security tool honestly, and it costs nothing to copy.
- "The maintainers decided, not us" is the part that makes the six findings mean something.
- It also means the zero-result stands: two frontier security products genuinely found nothing here.