What's this about?
Simon Willison famously tests every new model by asking it to draw "a pelican riding a bicycle" as SVG code. Dylan Castillo tested many animal×vehicle combinations and found some frontier models are conspicuously better at pelicans on bicycles than at otters on skateboards — suggesting labs may be quietly optimizing for the famous test.
What exactly is "benchmark contamination"?
When a test question becomes famous, it leaks into training data or gets deliberately practiced — so a high score stops meaning general ability and starts meaning "studied for this exact test." Like a student who memorizes last year's exam. The pelican test was useful precisely because it was absurd and unpracticed; fame is destroying its diagnostic value.
What's the community saying?
HN 413 points, 160 comments
- Willison himself showed up delighted: catching a lab specifically overfitting his pelican is "my absolute dream situation."
- A camp argues the test was always silly — making a text model emit SVG coordinates is like making an artist draw by dictating coordinates, and image models do this trivially.
- Odd finding: animal+plane combos underperform, possibly because models read "plane" as a typo for "plain."
- Meta-objection: the study uses an AI to judge AI drawings — who watches the judge?