VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Training

Researchers propose VRPO, a reinforcement learning framework that strengthens value modeling to handle noisy reward signals in large language model post-training. The approach uses auxiliary losses and information bottleneck techniques to enable value models to filter noise and generate more reliable advantage estimates, outperforming standard methods like PPO and GRPO across dialogue, math, and QA tasks.

VRPO addresses a fundamental challenge in applying reinforcement learning to large language models: reward signals from human feedback or automated systems are often incomplete, ambiguous, or contradictory. Traditional approaches treat the value model as a passive component that estimates expected returns, but VRPO repositions it as an active noise regulator that can correct and stabilize unstable reward signals. This represents a meaningful shift in how researchers think about the RL pipeline's architecture.

The technical contribution combines two complementary mechanisms: auxiliary losses derived from frozen language model properties (entropy and perplexity) that guide value modeling toward linguistically meaningful representations, and a variational information bottleneck that filters irrelevant information while preserving decision-critical features. This design prevents the value model from memorizing noise while maintaining sensitivity to genuine reward patterns.

The framework's consistent improvements across multiple task domains—multi-turn dialogue, mathematical reasoning, and science question answering—with both rule-based and learned reward models suggests the approach generalizes beyond specific reward configurations. This breadth is important because it indicates VRPO addresses a structural problem rather than optimizing for particular reward characteristics.

For the AI research community, this work emphasizes that robust policy optimization under realistic conditions requires reconsidering component roles within the learning pipeline. As LLM post-training becomes increasingly important for competitive performance, methods that handle imperfect supervision become more valuable. The research provides practical techniques that practitioners can implement without architectural overhauls, making adoption more feasible for existing systems.

• →VRPO repositions value models from passive predictors to active noise regulators in reinforcement learning pipelines.
• →Auxiliary losses from frozen language model properties improve value estimation stability under noisy reward supervision.
• →The framework consistently outperforms PPO and GRPO baselines across dialogue, reasoning, and QA tasks with multiple reward types.
• →Variational information bottleneck enables value models to filter noise while preserving decision-critical information.
• →Robust value modeling emerges as central to reliable policy optimization in real-world LLM post-training scenarios.