Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex

Researchers propose Listwise Policy Optimization (LPO), a new framework for training large language models that improves upon existing reinforcement learning approaches by explicitly projecting policies toward target distributions on the response simplex. The method demonstrates consistent performance improvements across reasoning tasks while maintaining training stability and response diversity.

This research addresses a fundamental challenge in post-training large language models: how to effectively optimize reasoning capabilities through reinforcement learning. The key innovation lies in revealing that existing group-based policy gradient methods implicitly define target distributions, then making this process explicit through direct projection onto the response simplex. By demystifying the optimization geometry, LPO enables more principled and controllable training dynamics.

The work builds on years of progress in RLVR (reinforcement learning with verifiable rewards), which has become essential for training reasoning-focused models. Previous approaches like group relative policy optimization operated through implicit targeting mechanisms that lacked clear optimization guarantees. LPO's explicit target-projection framework provides theoretical monotonic improvement bounds while offering practitioners flexibility in choosing divergence metrics that suit their specific training objectives.

For the AI development community, this represents incremental but meaningful progress in LLM optimization. The method's demonstrated improvements across diverse reasoning tasks and model architectures suggest practical applicability beyond academic settings. The emphasis on optimization stability and response diversity addresses real concerns in production training, where instability and mode collapse represent significant implementation challenges.

The framework's modular design—decoupling target specification from projection mechanisms—enables future extensions and hybrid approaches. Researchers and practitioners working on reasoning models should monitor adoption patterns and empirical results in real-world training scenarios. The stronger theoretical guarantees compared to baseline methods may influence how organizations design their post-training pipelines for capability-focused model development.

• →LPO makes explicit the implicit target distributions used by existing group-based policy gradient methods, improving optimization clarity and control.
• →The framework provides monotonic improvement guarantees with self-correcting projection gradients, addressing stability concerns in LLM training.
• →Empirical results show consistent performance improvements over baseline methods across multiple reasoning tasks and model scales.
• →Flexible divergence selection enables practitioners to tailor optimization dynamics to specific training objectives and constraints.
• →The decoupled projection architecture supports future extensions and hybrid optimization approaches for reasoning-focused LLMs.