The Two-Stage Decision-Sampling Hypothesis: Understanding the Emergence of Self-Reflection in RL-Trained LLMs

Researchers introduce the Two-Stage Decision-Sampling Hypothesis to explain how reinforcement learning enables self-reflection capabilities in large language models, demonstrating that RL's superior performance stems from improved decision-making rather than generation quality. The theory shows that reward gradients distribute asymmetrically across policy components, explaining why RL succeeds where supervised fine-tuning fails.

This research addresses a fundamental question in modern AI development: why do reinforcement learning-trained language models develop self-correction abilities that supervised fine-tuning cannot achieve? The authors propose a mechanistic framework decomposing LLM policies into sampling (generation) and decision (verification) components, revealing how different training objectives create distinct gradient distributions.

The theoretical contribution centers on the Gradient Attribution Property, which characterizes how reward signals propagate through model components. Crucially, surrogate rewards achieve balanced gradient attribution across both sampling and decision functions, while supervised fine-tuning with KL penalties creates unbalanced attribution that under-optimizes the decision-making component. This asymmetry explains why pure SFT produces weaker self-reflection despite using identical training data.

This mechanistic understanding has substantial implications for AI development practices. As language models scale and tasks increase in complexity, the ability to verify and revise outputs becomes increasingly valuable. The research suggests that practitioners seeking to develop reasoning-capable models should prioritize RL-based training methods, specifically those using surrogate rewards rather than standard supervised approaches. The empirical validation on arithmetic reasoning provides concrete evidence supporting the theoretical predictions.

The work advances beyond empirical observation toward causal understanding of emergent capabilities, establishing a foundation for deliberately engineering self-reflection in future models. This understanding could accelerate development of more reliable AI systems capable of autonomous error correction and verification, particularly valuable in high-stakes applications requiring trustworthy decision-making.

• →RL training achieves balanced gradient attribution across model components, while SFT creates unbalanced attribution favoring generation over verification
• →Self-reflection improvements in RL-trained models stem primarily from enhanced decision-making capabilities rather than better sampling or generation
• →KL penalties in SFT create asymmetric regularization that constrains sampling while leaving decision functions under-optimized
• →The Two-Stage Decision-Sampling Hypothesis provides a mechanistic explanation for why RL outperforms SFT on reasoning tasks
• →Understanding gradient distribution patterns enables more deliberate engineering of self-correction abilities in future language models