What's this about?
The day's top paper by a wide margin, and it asks the question every previous skills evaluation skipped. Existing work measures whether skills improve aggregate task success; this asks when they help, why they work, and where they fail. Through controlled experiments across multiple benchmarks, agent harnesses and models, the authors isolated the effects of representation, outcome annotation, retrieval difficulty and cross-framework robustness. They normalized 8,135 trial records and hand-coded 240 into a taxonomy of three categories and twelve skill-use modes. The central finding: procedural anchoring accounts for 65.7% of cases where skills help, versus 4.5% for explicit knowledge injection. Skills beat Workflow Memory by 6.06 points in matched comparisons. And retrieval is a separate bottleneck — as the skill pool grows from 5 to 100, actual-use precision falls from 29.6% to 3.3%.
What exactly is "procedural anchoring" versus "knowledge injection"?
These are two theories of why skills help, and the paper says one is mostly right. Knowledge injection assumes the skill supplies facts the model lacked — a reference document it reads. Procedural anchoring assumes the model already knew how, but its execution was unstable: it wandered, skipped steps, or improvised inconsistently. A skill then acts less like an encyclopedia and more like a checklist, holding the agent to one reliable path. The analogy is a surgical checklist: the surgeon isn't learning anatomy from it, they're being kept from skipping step four at 3am. That distinction matters because it tells you what to write. Skills should encode *procedure*, not information the model can already retrieve.
Why is this worth noticing?
- The retrieval collapse is the most actionable number. Precision dropping from 29.6% to 3.3% as a library grows from 5 to 100 skills means large skill collections actively hurt unless retrieval improves — a direct warning to the curated-skill-pack trend of the past fortnight, and the exact problem SkillZip was compressing for.
- A counterintuitive secondary result: confusable distractors hurt offline identification, yet downstream success stayed stable, and exact ground-truth invocation turned out to be *neither sufficient nor necessary*. Retrieving the "right" skill is not what determines whether the task succeeds.
- It's the empirical companion to StateM. Yesterday's top paper showed runbooks producing frontier results by stabilizing execution; this explains why that works, and puts a percentage on it.