Goldilocks RL: Tuning Task Difficulty to Escape Sparse Rewards for Reasoning

Researchers introduce Goldilocks, a curriculum learning strategy that improves reinforcement learning efficiency for language models by having a teacher model dynamically select training questions of optimal difficulty for the student model. This addresses the sample inefficiency problem in sparse-reward RL training and demonstrates performance gains on reasoning tasks compared to standard approaches.

The Goldilocks research tackles a fundamental challenge in training reasoning-capable language models: the extreme sample inefficiency of reinforcement learning with sparse rewards. Current methods require models to explore vast solution spaces with minimal feedback signals, making large-scale deployment computationally expensive and slow. The proposed solution adapts classical curriculum learning—the intuition that humans learn best from moderately challenging material—to modern large language model training.

This work builds on two converging trends in AI research. First, RL-based training has proven effective for unlocking reasoning in LLMs, exemplified by recent breakthroughs from major labs. Second, curriculum learning has re-emerged as a practical technique for improving training efficiency, though prior implementations failed to scale beyond small datasets. Goldilocks bridges this gap by using a teacher model to continuously assess student model performance and adaptively select questions that match the student's current capability level, neither trivial nor impossibly difficult.

The practical implications are substantial for AI developers and researchers. By improving sample efficiency under fixed compute budgets, Goldilocks reduces the computational cost of training reasoning models, potentially democratizing access to advanced model development. This is particularly relevant for resource-constrained organizations competing against well-funded labs with massive GPU clusters.

The validation on OpenMathReasoning demonstrates measurable improvements, suggesting the approach generalizes beyond toy problems. Future work should explore applicability across different reasoning domains and model scales. The adaptive nature of the strategy, which responds to individual student model progress, hints at personalized training protocols that could become standard practice in LLM development.

• →Goldilocks uses teacher-student dynamics to select optimally-difficult training examples, improving RL sample efficiency for language model reasoning.
• →The approach adapts classical curriculum learning principles to large-scale LM training, addressing scalability limitations of prior work.
• →Demonstrates measurable performance improvements on OpenMathReasoning dataset under identical compute budgets compared to standard GRPO training.
• →Reduces computational costs for training reasoning-capable models, potentially expanding accessibility for smaller research teams.
• →Teacher model continuously adapts to student progress, enabling dynamic difficulty calibration throughout training.