DGPO: Distribution Guided Policy Optimization for Fine Grained Credit Assignment

Researchers introduce Distribution Guided Policy Optimization (DGPO), a novel reinforcement learning framework that improves how large language models learn to perform complex reasoning tasks by assigning credit at the token level rather than sequence level. DGPO replaces unstable KL divergence penalties with bounded Hellinger distance and adds an entropy gating mechanism, achieving state-of-the-art performance on challenging math benchmarks like AIME2024 and AIME2025.

DGPO represents a meaningful advancement in reinforcement learning for large language model alignment, addressing fundamental limitations in how current methods like Group Relative Policy Optimization assign credit for reasoning steps. The core innovation lies in moving from coarse sequence-level evaluation to fine-grained token-level credit assignment, enabling the model to identify which specific reasoning steps drive correct answers versus which generate hallucinations or incorrect paths. This distinction matters significantly because chain-of-thought reasoning in mathematical problem-solving involves multiple sequential decisions, and penalizing entire sequences equally wastes learning signal on irrelevant tokens.

The technical contribution—replacing unbounded KL divergence with bounded Hellinger distance—addresses a practical problem: KL penalties create gradient instability and mode-seeking behavior that discourages exploration of novel reasoning trajectories. By reframing distribution deviation as a guidance signal rather than a penalty, DGPO enables safer token-level exploration without gradient explosion risks. The entropy gating mechanism adds sophistication by weighting deviations based on the model's epistemic uncertainty, thereby distinguishing genuine reasoning breakthroughs from random noise.

For the broader AI research community, DGPO's empirical results on AIME benchmarks demonstrate tangible improvements: 60.0% accuracy on AIME2024 and 46.0% on AIME2025 for Qwen2.5-32B substantially exceed previous baselines. These gains matter because mathematical reasoning serves as a key benchmark for evaluating language model capabilities. The framework's critic-free design—eliminating the need for a separate value network—also reduces computational overhead, making the approach more practical for researchers with limited resources. Industry applications in automated reasoning systems, educational AI, and research assistance benefit from more reliable reasoning alignment.

• →DGPO achieves token-level credit assignment in reinforcement learning, enabling models to identify pivotal reasoning steps in chain-of-thought generation
• →Replacing KL divergence with bounded Hellinger distance eliminates gradient instability while safely quantifying policy exploration
• →Entropy gating mechanism scales deviation signals by epistemic uncertainty to suppress hallucinations while encouraging genuine reasoning breakthroughs
• →State-of-the-art performance on AIME2024 (60.0%) and AIME2025 (46.0%) benchmarks outperforms competitive baselines like DAPO
• →Critic-free design removes computational overhead of additional value networks while improving fine-grained credit reallocation