3 articles tagged with #preference-aggregation. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
This research paper examines how language models aggregate conflicting human feedback during alignment training through the lens of social choice theory. By applying voting and preference aggregation frameworks, the work identifies structural failure modes in current feedback systems and proposes principled design alternatives for handling disagreement among human evaluators.
Researchers identify a critical flaw in standard RLHF (Reinforcement Learning from Human Feedback) pipelines: they collapse culturally and contextually diverse human preferences into single scalar rewards, potentially misaligning AI systems in pluralistic societies. A study of Malaysian annotators found that 79% of prompts contained multiple majority-supported valid responses that standard aggregation would discard, suggesting current alignment measurement fails to capture legitimate interpretive diversity.
Researchers present DecompR, a method to improve how large language models handle tasks with conflicting stakeholder preferences by separating utility estimation from aggregation. Traditional holistic LLM judges create unstable implicit weights that cause significant score variability, especially as stakeholder numbers increase; the proposed approach fixes weights based on query structure before scoring to eliminate candidate-dependent weight drift.
