What's this about?
As AI systems make more morally loaded decisions, one popular response is moral preference elicitation: poll people on hypothetical dilemmas and aggregate the answers into a policy. This paper argues that process is not neutral — developer choices exert an invisible hand over the result.
What exactly is "moral preference elicitation"?
It's the approach behind projects like MIT's Moral Machine, which asked millions of people what a self-driving car should do in trolley-problem scenarios. The appeal is democratic legitimacy: rather than engineers deciding ethics, you ask the public. The paper's objection is that the developers still decide everything that determines the answer — which dilemmas are posed, how they're worded, who is sampled, and how disagreement is aggregated. Aggregation especially: majority rule, average preference and consensus give different answers from identical responses.
Why is this worth noticing?
- It targets a method gaining traction precisely because it appears to remove developer judgment, when the judgment has only moved somewhere less visible.
- Framing effects in moral dilemmas are large and well-documented, so the wording of the poll can substantially determine the policy it produces.
- It pairs with last week's "No One to Blame" paper: both argue that the mechanisms proposed to close accountability gaps in AI can quietly relocate the discretion rather than remove it.