On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training

Researchers prove that supervised fine-tuning (SFT) and reinforcement learning (RL) cannot be decoupled during large language model post-training, as each method degrades the performance gains of the other. The theoretical findings, verified experimentally, challenge the widespread industry practice of alternating these two training approaches and suggest optimal RL duration exists to balance competing objectives.

This research addresses a fundamental assumption in modern language model development: that SFT and RL can be treated as independent optimization stages. The paper provides rigorous mathematical proof that decoupling is impossible regardless of training order. When RL follows SFT, reinforcement learning increases the cross-entropy loss from supervised training. Conversely, when SFT follows RL, supervised fine-tuning reduces the reward signals achieved by reinforcement learning. This non-decoupling phenomenon has significant implications for how AI labs structure their post-training pipelines.

The theoretical contribution extends beyond merely identifying the problem. Under certain conditions (PL-based analysis), the authors derive the optimal duration for RL training that maximizes reward improvement while minimizing SFT degradation. They also establish the threshold beyond which RL training becomes counterproductive to SFT objectives. These findings explain why leading reasoning models like o1 alternate between SFT and RL—not because decoupling is possible, but because the interleaving minimizes mutual interference.

For the AI development community, this research provides mathematical justification for existing practices while offering precision about optimal training schedules. Developers cannot simply optimize SFT independently then apply RL; instead, they must carefully manage the trade-offs between competing objectives. The experimental validation on Qwen3-0.6B demonstrates that the theoretical predictions translate to observable performance degradation, lending credibility to the framework. This understanding could lead to more efficient post-training protocols that reduce computational costs while maintaining model quality, though significant engineering work remains to implement these insights at scale.

• →SFT and RL objectives provably conflict regardless of which training phase occurs first, invalidating the assumption they can be independently optimized.
• →Optimal RL duration exists as a mathematical function that balances reward improvement against SFT performance degradation.
• →The non-decoupling threshold defines when RL training becomes counterproductive, providing quantitative guidance for training duration.
• →Experimental validation on Qwen3-0.6B confirms theoretical predictions of performance degradation from the interaction between training methods.
• →Modern reasoning models alternate SFT and RL not by choice but by necessity to minimize mutual interference between competing objectives.