MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following

Researchers propose MDP-GRPO, an improved reinforcement learning method that stabilizes group relative policy optimization for instruction-following tasks by addressing three fundamental instabilities in reward normalization. The technique achieves up to 5% improvement in constraint satisfaction on language models while maintaining general performance capabilities.

MDP-GRPO addresses a critical technical challenge in reinforcement learning for large language models: the instability that emerges when training on discrete, low-variance reward signals. Standard group relative policy optimization (GRPO) relies on z-score normalization to compare performance within training batches, but this approach breaks down when most samples receive identical rewards—a common scenario in multi-constraint instruction following where tasks either succeed or fail completely. The researchers identify three specific failure modes: amplification of negligible differences, inability to detect meaningful signal when reward means differ from individual scores, and complete gradient collapse when all samples receive identical rewards.

The proposed solution combines four complementary techniques drawing from information theory and behavioral economics. Multi-temperature sampling increases reward distribution diversity without changing the underlying reward structure. Dual-anchor advantages restore gradient signals by comparing against both group and individual baselines rather than relying solely on group statistics. Prospect-theoretic shaping incorporates Kahneman-Tversky insights to asymmetrically penalize constraint violations more heavily than rewards for compliance, reflecting real-world priorities. Asymmetric KL regularization prevents the policy from diverging excessively during unstable training periods.

This work has meaningful implications for AI researchers developing instruction-following systems, as it enables more reliable training with verifiable binary or discrete rewards. The improvements on Llama-3.2-3B suggest the method scales effectively to practical model sizes. Stability with small group sizes reduces computational requirements for training. The preservation of general knowledge capabilities on MMLU and ARC indicates the approach doesn't sacrifice breadth for constraint satisfaction. This represents incremental but important progress in making reinforcement learning from discrete feedback more robust and practical.

• →MDP-GRPO stabilizes training under discrete, low-dispersion rewards by addressing three specific failure modes in z-score normalization.
• →The method combines multi-temperature sampling, dual-anchor advantages, prospect-theoretic shaping, and asymmetric KL regularization for improved stability.
• →Achieves up to 5% improvement in strict constraint satisfaction on Llama-3.2-3B without degrading general knowledge benchmarks.
• →Enables stable convergence with smaller batch sizes, reducing computational overhead for instruction-following training.
• →Applicable to any multi-constraint instruction following task where rewards are discrete and most samples receive identical scores.