Hidden Consensus:Preference-Validity Compression in Human Feedback

Researchers identify a critical flaw in standard RLHF (Reinforcement Learning from Human Feedback) pipelines: they collapse culturally and contextually diverse human preferences into single scalar rewards, potentially misaligning AI systems in pluralistic societies. A study of Malaysian annotators found that 79% of prompts contained multiple majority-supported valid responses that standard aggregation would discard, suggesting current alignment measurement fails to capture legitimate interpretive diversity.

This research addresses a fundamental measurement problem in AI alignment that has gone largely unexamined. Standard RLHF assumes human feedback can be reduced to a single "correct" preference, but this assumption breaks down in culturally diverse contexts where multiple valid interpretations coexist. The Malaysia-based study reveals that single-winner aggregation methods systematically eliminate coherent, locally-grounded responses that reflect legitimate cultural, practical, or linguistic frames. This matters because training AI systems on artificially compressed preferences forces them to optimize for a false consensus rather than genuine alignment.

The broader context involves growing recognition that Western-centric AI development creates blind spots for non-Western users and values. As AI systems expand globally, alignment methods that assume universal preference hierarchies become increasingly problematic. The research demonstrates this isn't theoretical—empirical analysis shows majority-supported responses vanish in aggregation, and annotators frequently select multiple acceptable options, indicating the single-scalar approach fundamentally misrepresents human judgment.

For the AI industry, this challenges the current RLHF paradigm that powers models like ChatGPT and Claude. If alignment measurement is invalid for pluralistic societies, systems trained on compressed preferences may systematically underperform for non-Western users or fail to respect culturally grounded values. Developers relying on standard RLHF pipelines risk deploying systems that appear well-aligned but actually reflect preference collapse rather than true alignment. The proposed solution—Validity-Preserving Consistency—suggests future methods should maintain multiple valid response distributions rather than optimizing toward a single target.

• →Standard RLHF aggregation discards 79% of majority-supported alternative responses, treating cultural diversity as annotation noise rather than valid disagreement.
• →Single-scalar reward targets misrepresent plural-valid preferences and force false consensus in culturally heterogeneous contexts.
• →Current alignment measurement fails as a validity indicator for non-Western users and regionally grounded interpretations.
• →Multiple majority-supported responses appearing in the same prompt suggest AI alignment methods must preserve interpretive plurality rather than collapse it.
• →Future RLHF pipelines should satisfy Validity-Preserving Consistency to remain stable across different cultural and contextual frames.