Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning

Researchers have developed Curvature-Aware Policy Optimization (CAPO), a new algorithm that improves training stability and sample efficiency for Large Language Models by up to 30x. The method uses advanced mathematical optimization techniques to identify and filter problematic training samples, requiring intervention on fewer than 8% of tokens.

• →CAPO algorithm achieves up to 30x improvement in sample efficiency compared to standard GRPO for LLM reasoning tasks.
• →The method uses second-order geometry and curvature information to identify samples that cause unstable training updates.
• →CAPO requires minimal intervention, rejecting fewer than 8% of tokens during training.
• →The algorithm enables more aggressive learning regimes where baseline methods catastrophically fail.
• →Theoretical guarantees for monotonic improvement are established under realistic assumptions.