What's this about?
Static scanners are increasingly used to find executable or otherwise unsafe content inside machine-learning artifacts — pickled model files, serialised weights, anything that can carry code. This paper argues the standard metrics only describe the cases where a scanner produced a usable verdict at all, and proposes evaluating coverage and failure recovery instead.
Why is "coverage" different from accuracy?
Precision and recall measure how often a scanner is right *when it answers*. They say nothing about how often it silently declines to answer — an unsupported file format, a truncated archive, a nested container it cannot open, a parse error. Those cases usually get dropped from the denominator, so a scanner that handles 40% of your artifacts perfectly can post an excellent F1 score while leaving most of your surface unexamined. Failure recovery asks the follow-up question: when a scanner hits something it cannot parse, does it fail loudly or quietly?
Why does this matter to you?
- This is directly about how to evaluate the class of tool you are building — and specifically about the failure mode where a scanner looks good because it only grades itself on what it could read.
- "Fails loudly" is a design requirement it makes explicit: an unparseable artifact must surface as unknown, never as clean.
- It gives you defensible language for reporting coverage alongside findings, which is usually the honest answer to "did you scan everything?"