What's this about?
A study reported that when an AI assistant gave people wrong answers, they became *less* accurate on the task but *more* confident in their answers — the worst possible combination: AI advice suppressed critical thinking instead of augmenting it.
What exactly is "automation bias"?
The phenomenon underneath is automation bias — humans' tendency to over-trust a machine's output and stop double-checking, precisely because it usually is right. It predates LLMs (pilots and autopilot, doctors and diagnostic software), but chatbots supercharge it: they answer fluently and confidently even when wrong, and fluency reads as competence to human brains.
What's the community saying?
HN 282 points, 160 comments — and unusually, the top comments attack the study
- Methodology takedown — the most-upvoted technical reply argues nothing tested is AI-specific: the researchers gave people deliberately wrong answers; a wrong search result or wrong human would produce the same effect. Trivia questions also make it low-stakes.
- Lived-experience agreement anyway — advice subreddits are "going to shit" as people relay ChatGPT answers instead of first-hand knowledge; the study matches what moderators see daily.
- The pessimist synthesis — even if models get smarter, agreeable AI + confident humans = stupider ideas held more firmly, in exactly the domains where users can't tell.
- Sourcing complaints — the article's links don't actually lead to the study PDF, which HN treats as its own small indictment of the AI-news food chain.