RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning

Researchers propose POPO (Group Prioritized Off-Policy Optimization), a new framework that improves reinforcement learning for large language model reasoning by efficiently reusing ineffective training samples without computational overhead. The method addresses a critical limitation in RLVR systems where many training samples yield zero-variance rewards, enabling faster model improvement across mathematics, planning, and visual reasoning tasks.

The paper tackles a fundamental inefficiency in reinforcement learning for LLMs. Current RLVR approaches struggle with data quality—many sampled prompts generate response groups that are uniformly correct or incorrect, providing no learning signal. Existing solutions rely on expensive filtering through additional model rollouts, creating significant computational bottlenecks that limit practical adoption of RL-based reasoning improvements.

POPO introduces a two-component solution addressing this waste. The prioritized group replay mechanism intelligently substitutes ineffective on-policy samples with effective off-policy ones using a recency-based strategy that balances sample quality and off-policiness degree. Simultaneously, decoupled importance sampling corrects the statistical bias introduced by drawing from historical data while maintaining stable policy updates through trust-region constraints. This design avoids the systematic biases and suboptimal constraints plaguing existing efficiency approaches.

The framework's significance extends beyond academic optimization. For practitioners deploying LLM reasoning systems, POPO reduces computational requirements substantially—fewer rollouts translate to lower infrastructure costs and faster iteration cycles during model development. This democratizes access to advanced reasoning capabilities by making RL finetuning more resource-efficient. The empirical validation across diverse domains (mathematics, planning, geometry) demonstrates consistent improvements rather than domain-specific gains.

The broader implication affects how the AI community scales reasoning capabilities. As models grow larger, efficient training becomes paramount. POPO's approach of maximizing existing data utility rather than generating more data aligns with sustainability and efficiency trends in AI research. This methodology could influence how production systems handle the expensive process of improving model reasoning through reinforcement learning.

• →POPO eliminates computational overhead from filtering ineffective samples by intelligently replaying effective off-policy data instead.
• →Decoupled importance sampling mitigates off-policy bias while maintaining stable policy updates across diverse reasoning tasks.
• →Framework demonstrates substantial acceleration of RL finetuning with fewer model rollouts across mathematics, planning, and visual geometry domains.
• →Approach addresses a critical inefficiency where 50%+ of training samples generate zero-variance rewards providing no learning signal.
• →Method reduces infrastructure costs and iteration time for practitioners deploying large language model reasoning improvements.