AINeutralarXiv – CS AI · 9h ago6/10
🧠
VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training
Researchers introduce VESPO, a new method for training large language models using reinforcement learning that solves the variance problem in off-policy updates. The technique uses a principled mathematical approach to weight sequences rather than tokens, enabling stable training even when data becomes stale, with demonstrated improvements on math and code generation tasks.