Gradient Extrapolation-Based Policy Optimization

Researchers propose GXPO, a new policy optimization technique for reinforcement learning that approximates multi-step lookahead using only three backward passes instead of many, improving large language model reasoning performance by 1.65-5.00 points over standard GRPO while achieving up to 4x step speedup.

GXPO addresses a fundamental computational bottleneck in reinforcement learning for language models. While full multi-step lookahead produces superior policy updates, it requires prohibitively expensive repeated backward passes. This work bridges that gap through gradient extrapolation, predicting future optimization trajectories without actually computing them, making advanced optimization techniques practical for real-world training.

The technique emerges from the broader push to improve reasoning capabilities in large language models through RL, particularly for mathematical problem-solving. Prior approaches like GRPO accept single-step updates for computational efficiency, leaving performance gains on the table. GXPO's innovation lies in its stability mechanism—automatically reverting to standard GRPO when extrapolation becomes unreliable—ensuring robustness across diverse training scenarios.

For developers training reasoning models, GXPO reduces the computational cost of reaching peak performance by 4x in step count and 2.33x in wall-clock time, directly improving iteration speed and reducing infrastructure costs. The improvements across Qwen2.5 and Llama models suggest the technique generalizes well, making it potentially valuable for any organization fine-tuning language models on verifiable tasks like math reasoning or coding.

The mathematical foundation—the surrogate analysis explaining when extrapolation is exact—indicates this is not merely empirical trickery but a principled approach grounded in optimization theory. Future work likely explores application to broader reasoning domains and integration with other optimization methods.

• →GXPO approximates expensive multi-step lookahead using only three backward passes with gradient extrapolation
• →Performance improves 1.65-5.00 points over GRPO and 0.14-1.28 points over SFPO across math reasoning tasks
• →Achieves up to 4x step speedup and 2.33x wall-clock speedup to reach equivalent accuracy levels
• →Automatic stability detection reverts to single-pass GRPO when extrapolation signals become unreliable
• →Demonstrated effectiveness on Qwen2.5 and Llama models suggests broad generalization potential