VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training

Researchers introduce VESPO, a new method for training large language models using reinforcement learning that solves the variance problem in off-policy updates. The technique uses a principled mathematical approach to weight sequences rather than tokens, enabling stable training even when data becomes stale, with demonstrated improvements on math and code generation tasks.

VESPO addresses a fundamental technical challenge in LLM training: the instability that arises when models learn from data generated by older versions of themselves. Off-policy corrections in reinforcement learning typically rely on importance sampling, which measures how much a new policy differs from an old one, but this creates extremely high variance in autoregressive language generation where small policy shifts compound across token sequences.

The problem has grown more acute as organizations scale LLM training across multiple GPUs and TPUs, where generation inevitably lags behind policy updates. Existing solutions like PPO apply ad-hoc fixes—token-level clipping or sequence normalization—that reduce variance but introduce bias and lack theoretical grounding. VESPO derives a mathematically principled reshaping kernel from variational inference that directly bounds variance while operating on full sequences rather than individual tokens.

The empirical validation is substantial. Testing on mathematical reasoning and code generation shows VESPO maintains stability under extreme conditions (staleness up to 64x) while outperforming recent alternatives across both dense and mixture-of-experts architectures. This matters because it reduces engineering overhead and enables practitioners to use longer training horizons without quality degradation.

For the AI development community, this represents progress toward more reliable training infrastructure. The open-sourced code accelerates adoption among teams building reasoning-heavy models. The theoretical contribution—explicit variance bounds on the reshaping kernel—also provides guidance for future off-policy RL work beyond language models, though practical deployment impact depends on whether major labs adopt this over existing PPO variants.

• →VESPO introduces a principled mathematical approach to variance reduction in off-policy LLM training without relying on heuristic engineering tricks.
• →The method maintains stable training even with severe data staleness (64x), enabling more efficient distributed training pipelines.
• →Sequence-level reshaping outperforms token-level clipping on both dense and mixture-of-experts models in math and code generation tasks.
• →Explicit variance bounds on the reshaping kernel provide theoretical guarantees that prior methods lack.
• →Open-source implementation enables rapid adoption across the LLM training community.