sGPO: Trading Inference FLOPs for Training Efficiency in RLVR

Researchers introduce sGPO (sorted Group Policy Optimization), a training method that reduces computational waste in reinforcement learning by using cheap inference to profile query difficulty and dynamically allocate training resources. The approach achieves 3x reduction in total training compute while maintaining or improving performance, representing a significant efficiency breakthrough for large-scale AI model training.

sGPO addresses a fundamental inefficiency in RLVR training pipelines where fixed rollout budgets fail to account for varying query difficulty relative to current policy capability. Current systems waste substantial computational resources on both trivially easy queries (where policies already succeed) and impossibly hard queries (where policies never succeed), neither contributing meaningful learning signals. This symmetric waste represents a critical bottleneck in scaling reinforcement learning systems.

The innovation leverages an asymmetric cost structure: inference compute is significantly cheaper than training compute. By performing a single profiling pass with small parallel sample batches under the initial policy, sGPO obtains empirical success rates that serve as query-difficulty proxies. This enables three simultaneous optimizations—filtering trivial queries, downsampling intractable ones, and constructing curricula ordered by difficulty—all from one offline computation. The rule-based allocation (setting rollout group size to inverse success rate) is elegantly simple yet captures the core optimization objective.

For the AI training industry, a 3x reduction in training compute translates directly to dramatic cost savings, faster iteration cycles, and reduced environmental impact. This matters particularly for organizations training large language models and reasoning systems where RLVR is increasingly central to capability development. The approach scales across different model sizes and problem domains, making it broadly applicable rather than domain-specific.

The practical impact extends beyond efficiency metrics: faster training cycles enable more experimental iterations, lower barriers to entry for resource-constrained labs, and accelerated progress in AI capability development. Future work likely involves combining sGPO with other efficiency techniques and extending the methodology to multi-task and open-ended learning scenarios.

• →sGPO reduces total training compute by 3x by using cheap inference profiling to identify and filter easy/hard queries before training
• →The method trades small inference costs for large training efficiency gains by dynamically allocating rollout budgets based on query-specific success rates
• →Simultaneous data filtering, adaptive allocation, and curriculum learning emerge naturally from a single profiling pass without explicit design
• →Performance matches or exceeds baselines while dramatically reducing computational waste from unsolvable and trivial queries
• →Approach is broadly applicable across domains and model sizes, with direct implications for training cost reduction and environmental impact