What's this about?
Evaluating the hidden costs of personalization in large language models. 24 upvotes.
What could personalization cost?
Three things, roughly. Capability — a model tuned toward your preferences may get worse at things you have not expressed preferences about. Epistemics — a model that has learned what you like is a model that has learned what you will agree with, which is the mechanism behind sycophancy. And privacy — personalization requires retaining a model of you, which is data that exists, can be requested, and can leak. The paper's framing as "hidden" is the point: these costs are not what personalization is marketed on, and they are not what gets measured.
Why does this matter to you?
- It is directly relevant to the ChatGPT Ads mechanics covered yesterday, where broader-context personalization is opt-in-ish and user-controllable.
- The sycophancy mechanism is the one with the clearest path to harm and the least measurement.
- Memoryfields' critique from yesterday rhymes: lab memory systems store memories about *you* when information about the world would be more useful.