From $\boldsymbol{\log\pi}$ to $\boldsymbol{\pi}$: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight

Researchers introduce Decoupled Gradient Policy Optimization (DGPO), a new reinforcement learning method that improves large language model training by using probability gradients instead of log-probability gradients. The technique addresses instability issues in current methods while maintaining exploration capabilities, showing superior performance across mathematical benchmarks.

• →DGPO uses probability gradients rather than log-probability gradients to solve training instability issues in large language models.
• →The method resolves the conflict between stability and exploration in reinforcement learning with verifiable rewards.
• →Testing across DeepSeek-R1-Distill-Qwen models (1.5B/7B/14B parameters) showed consistent outperformance on mathematical benchmarks.
• →Traditional hard clipping methods stifle exploration while soft clipping methods suffer from divergent weights as probabilities approach zero.
• →The decoupled decay mechanism uses asymmetric, continuous decay based on importance sampling ratios for boundary tokens.