ESPO: Early-Stopping Proximal Policy Optimization

Researchers propose ESPO, an optimization technique that improves large language model training by detecting and terminating failed reasoning trajectories early rather than forcing completion. The method reduces computational waste by over 20% while achieving superior performance on mathematical reasoning benchmarks compared to standard PPO training.

ESPO addresses a fundamental inefficiency in reinforcement learning for language models: wasted computation on trajectories that fail early. When an LLM makes a reasoning error partway through a problem, current algorithms continue generating tokens that never receive positive feedback, squandering computational resources and corrupting gradient signals with post-failure noise. This represents a real economic and environmental concern given the enormous token volumes processed during model training.

The technique works by monitoring cumulative regret during generation and terminating rollouts when failure becomes apparent, then treating truncated sequences as absorbing states with terminal rewards. This concentrates learning signals efficiently near actual failure points without requiring additional reward models or manual annotation, making it practically implementable. The approach reflects broader trends in efficient AI training, where techniques like compute-optimal scaling and selective token generation have gained prominence.

On DeepSeek-R1-Distill-Qwen-7B, ESPO demonstrates measurable gains across multiple benchmarks—46.28% on AIME versus 45.25% for PPO—while reducing overall rollout tokens by 20% or more. These improvements matter for both research and production settings, as they indicate that better algorithmic design can substitute for raw compute. For the AI industry, such efficiency gains compound at scale, potentially reducing training costs and environmental impact significantly.

The work suggests that early-stopping mechanisms warrant deeper investigation in LLM training pipelines. Future applications might extend these principles to other domains where trajectory failure is detectable, and competitive implementations may emerge across major labs building reasoning models.

• →ESPO detects failed reasoning steps early and terminates rollouts, eliminating wasted computation on non-rewarded tokens.
• →The method surpasses standard PPO on mathematical reasoning benchmarks while reducing cumulative rollout tokens by over 20%.
• →No additional reward models or human annotation required, making the approach practical for production training.
• →Early-stopping regret monitoring concentrates TD errors efficiently near failure points without post-failure noise pollution.
• →Results demonstrate that algorithmic efficiency can deliver performance gains comparable to increased compute budgets.