What's this about?
A paper finding that LLMs fabricate user profiles — inventing traits and preferences the conversation never supported — and, worse, that asking the model to self-monitor for this makes it *more* misleading, not less (18↑).
What exactly is a "fabricated user profile"?
When an assistant infers things about you to personalize its answers ("you seem like an expert, so..."), it can simply make those inferences up, then act on them confidently. The self-monitoring twist: telling the model to check its own assumptions produces plausible-sounding justifications for the fabrications rather than catching them.
Why does it matter?
It's a direct caution for every "it learns you" product (and pairs with yesterday's PAST-Bench). Personalization that invents the person is worse than no personalization, and the finding that self-monitoring backfires undercuts the standard "just ask the model to check itself" fix.