Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models

Researchers propose a mid-training technique using self-generated data to improve reinforcement learning in large language models. By exposing models to multiple problem-solving approaches before RL training, the method demonstrates consistent improvements across mathematical reasoning, code generation, and narrative tasks.

This research addresses a fundamental challenge in training language models: the diversity of reasoning approaches available during the learning process. Traditional RL training pipelines often rely on limited training data that may not expose models to the full range of problem-solving strategies applicable to complex tasks. The researchers tackle this gap by implementing a bootstrapped data-generation framework inspired by George Polya's classical problem-solving methodology, creating multiple valid solution paths for each training question during a mid-training phase before applying reinforcement learning optimization.

The theoretical contribution centers on explaining how policy-gradient updates can incentivize models to combine different reasoning approaches, creating a more robust foundation for subsequent RL fine-tuning. This represents an incremental but meaningful advance in understanding how model initialization quality impacts downstream performance. The empirical validation spans multiple domains—mathematical reasoning benchmarks, code generation, and narrative reasoning tasks—suggesting the approach generalizes beyond narrow use cases.

For the AI development community, this work has practical implications for training pipelines. Organizations developing LLMs could incorporate similar data-augmentation strategies during mid-training stages without requiring architectural changes or additional computational overhead beyond standard fine-tuning. The approach demonstrates that thoughtful curriculum design and data diversity during intermediate training phases can compound improvements achieved through subsequent RL optimization. The research provides both theoretical grounding and practical evidence that exposure to diverse reasoning strategies earlier in training yields measurable downstream benefits, potentially influencing how practitioners structure their model development workflows.

• →Mid-training on self-generated diverse data improves subsequent reinforcement learning performance in language models
• →The bootstrapped data-generation framework leverages Polya's problem-solving methods to create multiple solution variants
• →Improvements are demonstrated across mathematical reasoning, code generation, and narrative reasoning benchmarks
• →Policy-gradient updates can effectively incentivize combining multiple problem-solving approaches within a single model
• →The technique provides a practical, non-architectural improvement to existing LLM training pipelines