What's this about?
kimmonismus argued that Qwen 27B is the "DeepSeek moment" for open source: it matches closed-source state-of-the-art from a few months ago and runs on a single RTX 5090, which he called a game changer without exaggeration. Ethan Mollick tested it and disagreed sharply — it's a genuinely good local model, but "immediately, absolutely and obviously nowhere near as good" as frontier models on agentic tasks and complex work. His advice: do your own benchmarking rather than reading a leaderboard row.
What exactly is the gap between benchmark scores and agentic performance?
Most benchmarks are single-turn: a question in, an answer out, graded once. Agentic work is long-horizon and compounding — the model plans, calls tools, reads results, recovers from errors, and stays coherent over dozens of steps. Small errors that barely register on a single-turn benchmark compound catastrophically over a long trajectory. So a model can genuinely match a frontier model on knowledge and reasoning questions while being far worse at doing a two-hour task, because what it lacks is reliability rather than intelligence. That gap is exactly what harness work like StateM exists to close.
What's the community saying?
- The disagreement is the useful content. Both are true: strong on benchmarks, weak on agentic work — and which matters depends entirely on what you're doing.
- "Do your own benchmarking" is the operative advice, especially given that the composite indices these claims rest on don't measure long-horizon reliability well.
- It connects to Jason Wei's argument from yesterday. He claimed internalized knowledge beats tool-mediated retrieval; this is the same gap observed from the deployment side — a small model that answers well but executes poorly.
- The RTX 5090 framing is what makes it feel like a moment, because "runs on hardware an enthusiast owns" is a different claim from "scores well," even if the second is what's being measured.