Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework

Researchers have developed a new preference learning framework that addresses bias in AI alignment by ensuring policies reflect true population distributions rather than just majority opinions. The approach uses social choice theory principles and has been validated on both recommendation tasks and large language model alignment.

• →Conventional preference learning methods create bias by prioritizing widely-held opinions over minority perspectives.
• →The new framework infers evaluator population distributions from pairwise comparison data to achieve proportional alignment.
• →The approach satisfies key axioms including monotonicity, Pareto efficiency, and population-proportional alignment.
• →A soft-max relaxation method allows trading off between population-proportional alignment and Condorcet winner selection.
• →The method has been successfully tested on tabular recommendation tasks and large language model alignment.