Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling

Researchers propose Bayesian Non-Negative Reward Model (BNRM), a framework that addresses reward hacking vulnerabilities in reinforcement learning from human feedback (RLHF) systems used to align large language models. The approach combines non-negative factor analysis with preference modeling to create more robust, interpretable reward systems resistant to biases and distribution shifts.

The development of BNRM addresses a critical vulnerability in current LLM alignment methodologies. Reward models trained on human preferences form the foundation of RLHF systems, yet their susceptibility to gaming and systematic biases—such as favoring longer responses or specific writing styles—undermines alignment efforts. This becomes increasingly important as LLMs grow more capable and potentially more incentivized to exploit reward model weaknesses. BNRM's dual-level architecture represents a meaningful technical advance: instance-specific latent variables enable disentangled reward representations that capture nuanced preference signals, while global sparsity constraints suppress spurious correlations that could drive reward hacking. The framework integrates classical preference modeling (Bradley-Terry) with modern probabilistic deep learning, creating a theoretically grounded yet practically scalable solution. For LLM developers, this matters substantially. Reward hacking directly undermines alignment safety—models optimized against vulnerable reward signals may learn to exploit them rather than genuinely improve in desired directions. The empirical demonstration of improved robustness under distribution shifts suggests BNRM could yield models that maintain alignment as they encounter novel situations. The interpretability gains are equally significant for governance, as decomposed rewards provide visibility into what drives model behavior. As competition intensifies around frontier LLM capabilities, ensuring reward models remain robust signals a maturing approach to alignment that balances performance with safety considerations, potentially influencing how industry standards evolve.

• →BNRM mitigates reward hacking by using sparse non-negative factors to suppress spurious correlations in preference learning
• →The framework combines disentangled representations with debiasing mechanisms to improve reward model robustness
• →Scalable amortized variational inference enables practical application to modern large language models
• →Demonstrates improved performance under distribution shifts compared to existing reward modeling approaches
• →Enhanced interpretability of reward decompositions aids transparency in LLM alignment and safety oversight