Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning

Researchers introduce Slow-Fast Policy Optimization (SFPO), a new reinforcement learning framework that improves training stability and efficiency for large language model reasoning. SFPO outperforms existing methods like GRPO by up to 2.80 points on math benchmarks while requiring up to 4.93x fewer rollouts and 4.19x less training time.

• →SFPO addresses instability issues in early training of LLM reasoning by decomposing updates into three stages: fast inner steps, repositioning, and slow correction.
• →The framework is plug-compatible with existing policy-gradient pipelines, requiring no changes to objectives or rollout processes.
• →SFPO achieves up to 2.80 point improvement over GRPO on math reasoning benchmarks.
• →Training efficiency is significantly improved with up to 4.93x fewer rollouts and 4.19x reduction in wall-clock time.
• →The reposition-before-update design helps control off-policy drift while maintaining training stability.