One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models

Researchers identified persistent biases in high-quality language model reward systems, including length bias, sycophancy, and newly discovered model-style and answer-order biases. They developed a mechanistic reward shaping method to reduce these biases without degrading overall reward quality using minimal labeled data.

• →Five state-of-the-art reward models still exhibit significant biases including length, sycophancy, and overconfidence issues despite prior mitigation efforts.
• →New bias categories were discovered related to model-specific writing styles and answer ordering preferences.
• →A post-hoc intervention method called mechanistic reward shaping was developed to mitigate low-complexity biases from spurious correlations.
• →The proposed solution reduces targeted biases while maintaining reward quality and generalizes to out-of-distribution scenarios.
• →The method is extensible to address new bias types as they are discovered in language model reward systems.