AI Alignment From Social Choice Perspectives

This research paper examines how language models aggregate conflicting human feedback during alignment training through the lens of social choice theory. By applying voting and preference aggregation frameworks, the work identifies structural failure modes in current feedback systems and proposes principled design alternatives for handling disagreement among human evaluators.

The alignment of large language models represents one of AI's most pressing challenges, particularly when training data reflects diverse and often conflicting human values. This paper reframes the feedback aggregation problem—how models learn from multiple human judges with different preferences—as a social choice problem analogous to voting systems. This perspective is valuable because current approaches typically treat disagreement as noise rather than meaningful signal, leading to models that may learn arbitrary or unstable objectives when human judges diverge.

The social choice framework brings decades of established theory to bear on AI alignment. Just as voting systems must navigate Arrow's impossibility theorem and strategic voting concerns, feedback aggregation faces analogous mathematical constraints. When multiple human evaluators rate model outputs differently, naive averaging or majority voting approaches can produce counterintuitive results that satisfy no evaluator's preferences. The paper identifies these failure modes systematically, revealing that certain design choices in feedback collection inadvertently violate desirable properties like consistency or fairness.

For AI developers and organizations deploying large language models, this work has immediate practical implications. Current industrial approaches to RLHF (reinforcement learning from human feedback) often optimize for convenience rather than principled preference aggregation, potentially embedding subtle biases or instabilities into production systems. The research expands the design space for handling disagreement, offering alternatives that explicitly acknowledge conflicting values rather than suppressing them. This matters particularly for applications spanning diverse user populations or cultural contexts where uniform preferences don't exist. The framework enables more transparent decision-making about whose values get prioritized when trade-offs are unavoidable.

• →Social choice theory reveals hidden failure modes in standard feedback aggregation methods used for LLM alignment.
• →Current RLHF approaches often treat human disagreement as noise rather than meaningful preference conflicts requiring principled resolution.
• →Mathematical impossibility results from voting theory constrain what any feedback aggregation system can simultaneously achieve.
• →The framework enables explicit design choices about value prioritization rather than implicitly embedding biases through naive averaging.
• →Structured aggregation methods can improve model robustness across diverse user populations with conflicting preferences.