What's this about?
This paper (12 upvotes) argues that standard LLM evaluation creates an "illusion of capability," because it only tests models under nominal conditions where they walk a narrow, highly optimized generation corridor. Real deployments push them out of it — complex system prompts, safety guardrails, unusual constraints — and the paper introduces a diagnostic stress test that operates at the decoding level.
What exactly is "decoding"?
Decoding is the step where a model turns its internal probability estimates into actual words: at each position it has a ranked list of candidate tokens and a strategy for choosing among them. Constrain that choice — forbid certain words, force a format, block a topic — and you're intervening at the decoding level. The paper's "taboo" test forbids specific tokens and observes whether the model still reasons coherently or falls apart, which is a proxy for how it behaves under the guardrails production systems actually impose.
Why is this worth noticing?
- It targets the gap between benchmark scores and deployment behavior, which is one of the most persistent complaints practitioners have about model evaluation.
- Because it operates at decoding rather than through prompting, it tests something prompt-based jailbreak and robustness evaluations structurally cannot reach.