ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillation

Researchers introduce ReNIO, a novel technique for improving large language model distillation by reweighting negative trajectories—incorrect reasoning paths generated by student models. The method shows that training on wrong outputs outperforms correct ones, and ReNIO leverages probability ratios to identify pivotal failure points without requiring full answer verification, delivering up to 10% improvements on mathematical reasoning benchmarks.

ReNIO addresses a fundamental inefficiency in on-policy distillation, a training approach where student models learn from their own outputs. The key insight challenges conventional wisdom: incorrect model outputs contain more valuable learning signals than correct ones because they preserve exploratory reasoning near the model's capability boundaries. This finding has significant implications for how AI teams optimize model training efficiency and resource allocation.

The mechanism behind ReNIO is elegant in its simplicity. Rather than observing final answer correctness—which requires expensive full rollouts—the method uses probability ratios between student and teacher models to identify tokens where divergence occurs. These divergence points signal likely reasoning failures and receive higher training weights. This prefix-only approach preserves computational advantages over full-rollout reinforcement learning while capturing the benefits of negative examples.

For the AI development community, ReNIO represents a meaningful optimization in model training pipelines. The technique applies across both mathematical reasoning and code generation, with consistent gains across different model sizes. The 8-10% improvements on benchmarks translate to either faster convergence during training or better performance with equivalent computational budgets—both valuable in an era of escalating training costs.

The practical implications extend beyond academia. As organizations deploy smaller distilled models for inference efficiency, ReNIO's improvements become increasingly important for maintaining reasoning quality at scale. The availability of open-source code democratizes the technique, enabling broader adoption across research and production environments.

• →Training on incorrect student-generated outputs consistently outperforms correct-only training in on-policy distillation setups.
• →ReNIO identifies high-value negative trajectories using student-to-teacher probability ratios without observing final answer correctness.
• →The technique achieves 8-10% relative improvements on mathematical reasoning benchmarks across multiple model sizes.
• →Prefix-conditioned probability analysis preserves computational efficiency compared to full-rollout reinforcement learning approaches.
• →Method applies successfully to both mathematical reasoning and code generation tasks, demonstrating broad applicability.