Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models

Researchers propose Dynamics-Predictive Sampling (DPS), a new method that improves reinforcement learning finetuning of large language models by predicting which training prompts will be most informative without expensive computational rollouts. The technique models each prompt's learning progress as a dynamical system and uses Bayesian inference to select better training data, reducing computational overhead while achieving superior reasoning performance.

• →DPS reduces computational overhead in RL finetuning by predicting informative prompts without expensive rollouts.
• →The method models prompt learning progress as a dynamical system with hidden Markov models.
• →Bayesian inference on historical reward signals enables efficient prompt selection for training.
• →Empirical results show DPS accelerates training and improves reasoning performance across mathematics, planning, and visual geometry tasks.
• →The approach addresses a key bottleneck where rollout computation can exceed the cost of the finetuning process itself.