What's this about?
Quesma's engineering post-mortem of building a custom deep-research pipeline, with the self-aware title as thesis: most clever token-optimization schemes cost more than they save, in engineering time and often in literal tokens.
What exactly is prompt caching (and why do "savings" backfire)?
LLM APIs charge per token, but offer prompt caching: if the beginning of your prompt is byte-identical to a recent request, the provider reprocesses it at a steep discount. The trap the HN thread names: most token-saving tricks — rotating retrieved context in and out, trimming history dynamically — change the prompt's prefix every turn, killing the cache. You save 10% of tokens and pay 3x for the ones that remain. Stable-prefix design often beats clever trimming.
What's the community saying?
HN 98 points, 131 comments — practitioners comparing scars
- The cache-killer insight — the thread's most-cited comment: "most 'token saving' ideas are cache killers"; anything making the prefix dynamic can cost more than it saves.
- Cheap-first funnels — start research passes with cheap models, escalate findings through increasingly powerful ones; starting in the right part of the search space beats any compression trick.
- Local-first hybrids — run the bulk on local models, reserve frontier APIs for the final 10%.
- "No hallucinations" skepticism — the article's claim draws the reliable objection that you can't rule-out hallucinations with rules or judge-models.
- Meta-snark — cloud AI blogging about using cloud AI to make cloud AI cheaper, for products not yet shipped.