Outcome-Based RL Provably Leads Transformers to Reason, but Only With the Right Data

Researchers prove that Transformers trained with reinforcement learning and outcome-based rewards spontaneously develop chain-of-thought reasoning capabilities, but only when training data includes sufficient 'simple examples' requiring fewer reasoning steps. The findings bridge theory and practice, explaining how sparse reward signals drive emergence of interpretable algorithmic behavior in language models.

This research addresses a fundamental question in AI development: how do neural networks discover systematic reasoning strategies when trained only on final outcomes? The authors provide rigorous theoretical proof that policy gradient optimization naturally guides Transformers toward structured, step-by-step reasoning on graph traversal tasks—mirroring real-world chain-of-thought patterns observed in large language models. The critical discovery centers on data distribution: the presence of simpler training examples acts as a learning scaffold, enabling models to generalize to more complex problems. Without sufficient mass on these easier instances, the same optimization fails entirely.

This work builds on empirical observations that outcome-based RL produces reasoning behavior in language models, adding mathematical rigor that was previously missing. The theoretical framework reveals why curriculum learning intuitively works—simple examples provide stable gradients that bootstrap discovery of generalizable strategies. The authors validate their findings across synthetic benchmarks and mathematical reasoning tasks with real models, demonstrating that theoretical insights transfer to practical settings.

For the AI development community, these findings clarify training dynamics that practitioners can leverage for more efficient model development. The emphasis on data distribution properties suggests that success in outcome-based RL depends critically on careful dataset construction, not just optimization algorithms. Understanding these mechanisms accelerates the path toward more interpretable and reliable AI systems capable of complex reasoning.

Future work should explore how these principles scale to larger models and more complex domains, and whether similar distributional requirements apply across different reasoning types.

• →Policy gradient RL provably drives Transformers to develop chain-of-thought reasoning when trained on final-answer correctness alone.
• →Simple examples in training data are essential for reasoning emergence; their absence makes learning infeasible.
• →Theoretical framework explains why curriculum learning and data composition critically matter in outcome-based RL.
• →Findings validated on both synthetic tasks and real language models on mathematical reasoning benchmarks.
• →Results suggest interpretable algorithmic strategies emerge naturally from sparse reward signals under proper distributional conditions.