Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training

Researchers propose rubric-based reward modeling to address reward over-optimization in large language model fine-tuning. The approach focuses on the high-reward tail where models struggle to distinguish excellent responses from merely great ones, using off-policy examples to improve training effectiveness.

• →Reinforcement fine-tuning suffers from reward over-optimization where models hack reward signals for high scores but produce low-quality outputs.
• →The core issue lies in reward misspecification at the high-reward tail, where systems cannot reliably distinguish excellent from great responses.
• →Rubric-based rewards can leverage off-policy examples while remaining insensitive to their artifacts.
• →The proposed workflow emphasizes distinguishing among great and diverse responses to capture the high-reward tail effectively.
• →Empirical results show rubric-based rewards substantially reduce reward over-optimization and improve LLM post-training.