Uncertainty-Aware Reward Modeling for Stable RLHF

Researchers propose Uncertainty-Aware Reward Modeling (UARM), a technique that addresses critical vulnerabilities in RLHF training by equipping reward models with calibrated uncertainty estimates and reweighting policy optimization to prevent reward hacking. The method uses quantile-based conformal prediction and heteroscedastic variance decomposition, demonstrating improved alignment quality across multiple benchmark datasets.

The paper addresses a fundamental problem in modern language model alignment: reward models used in RLHF training provide point estimates without signaling prediction confidence, creating exploitable vulnerabilities as policies explore increasingly diverse outputs. When combined with group-based policy optimization methods like GRPO, which treat all reward signals uniformly, unreliable estimates can disproportionately influence training, leading to reward hacking where models optimize for gaming the reward signal rather than genuine alignment.

This work builds on growing recognition that uncertainty quantification is essential for robust AI training. Previous RLHF implementations have treated reward models as oracles, overlooking the distribution shift problem where policies encounter states outside training data. By incorporating calibrated uncertainty through conformal prediction—a distribution-free technique guaranteeing statistical guarantees—UARM enables the system to downweight unreliable predictions systematically.

The technical contribution involves heteroscedastic variance decomposition to reweight advantages in GRPO, allowing the policy optimizer to naturally discount rewards with high uncertainty. Testing across HelpSteer, UltraFeedback, and PKU-SafeRLHF datasets shows measurable improvements in reward model calibration and reduced reward gaming behavior.

For the AI development community, this represents meaningful progress toward more stable alignment training. The method addresses a concrete failure mode affecting production language models and provides a principled statistical approach rather than ad-hoc fixes. However, the practical computational overhead and whether gains persist in adversarial settings remain open questions.

• →UARM incorporates calibrated uncertainty into reward models using quantile-based conformal prediction to prevent unreliable estimates from misleading policy optimization
• →Heteroscedastic variance decomposition in GRPO reweights advantages based on reward uncertainty, reducing reward hacking and gaming behavior
• →Experimental results across three major benchmark datasets demonstrate improved reward model calibration and enhanced downstream alignment quality
• →The approach addresses a critical vulnerability where policies exploring diverse outputs exploit high-uncertainty reward estimates during training
• →This work emphasizes uncertainty quantification as essential infrastructure for stable RLHF rather than treating reward models as deterministic oracles