PAFO: Pareto Fairness Optimization for Personalized Reward Modeling

Researchers propose PAFO, a Pareto fairness optimization framework that addresses bias in personalized reward models for large language models by improving performance for under-served user preference groups without degrading majority groups. The method uses group-specialized models and conditional margin-level supervision to create fairer LLM alignment across diverse user populations.

The research identifies a critical limitation in current personalized reward modeling: training data imbalance causes reward models to systematically favor users whose preferences appear frequently in training sets, effectively marginalizing minority preference groups. This personalized reward bias represents a fairness problem that extends beyond typical machine learning considerations—it directly impacts how AI systems serve heterogeneous populations.

Personalized reward models have become essential infrastructure as LLMs attempt to align with diverse user expectations. However, the reality of training data often reflects skewed preference distributions in real-world deployments. PAFO addresses this by formulating the problem as Pareto optimization, where improvements for minority groups need not sacrifice majority group performance. The technical approach—training separate models for different groups then distilling their boundaries into a unified model—allows inference-time simplicity while maintaining fairness gains.

For developers and organizations building personalized AI systems, this work carries practical significance. Fair reward modeling directly influences user satisfaction and system adoption across different demographic segments. Biased reward models can amplify existing inequalities, reducing utility for minority preference groups and potentially creating reputational or regulatory risks. The experimental validation on standard benchmarks demonstrates measurable improvements in both minority and majority group accuracy alongside reduced unfairness metrics.

Looking forward, the challenge lies in adoption. While PAFO requires no explicit group labels at inference, it demands group identification during training—a requirement that may conflict with privacy considerations in some deployments. Future research should explore how group fairness techniques scale with increasing numbers of preference categories and whether similar approaches apply to other alignment mechanisms beyond reward modeling.

• →PAFO mitigates personalized reward bias by optimizing for Pareto fairness across majority and minority user preference groups
• →The framework trains group-specialized models then distills them into a single unified model requiring no explicit group labels at inference
• →Experiments demonstrate simultaneous improvements in minority-group accuracy and majority-group performance while reducing user-level unfairness
• →Training data imbalance in reward models systematically disadvantages users with less common preferences in the population
• →Fair reward modeling addresses both practical utility and potential regulatory concerns in personalized AI system deployment