ADWIN: Adaptive Windows for Horizon-Aware On-Policy Distillation

ADWIN is a new framework for on-policy distillation that optimizes training efficiency by adaptively adjusting rollout lengths instead of requiring full completions for every update. The method reduces training costs by up to 4.1x while maintaining or improving accuracy on math and code reasoning tasks by identifying when shorter teacher-anchored sequences contain sufficient signal for learning.

ADWIN addresses a fundamental inefficiency in on-policy distillation training where every parameter update requires expensive full-rollout completions from the student model. This constraint becomes particularly wasteful when student trajectories remain well-aligned with teacher preferences early on, yet continue to drain compute resources through unnecessary later-stage generation. The framework reframes rollout length as a dynamic decision problem rather than a fixed hyperparameter, using delayed full-rollout probes to continuously audit when short prefixes capture sufficient learning signal.

The research builds on growing recognition that not all tokens in a sequence contribute equally to model improvement. Earlier work identified this phenomenon in supervised fine-tuning contexts, but ADWIN specifically targets the on-policy setting where this inefficiency compounds due to the cost of generating student trajectories. By implementing staleness control mechanisms, the framework prevents the adaptive windowing strategy from becoming stale as the student model evolves, ensuring prefix-level training remains reliably aligned with full-rollout objectives.

For the machine learning systems community, ADWIN demonstrates significant practical value across diverse reasoning domains. The 4.1x training cost reduction without accuracy loss has direct implications for model development budgets and accessibility. This efficiency gain matters especially for resource-constrained teams and smaller organizations scaling reasoning models. The framework's compatibility with both single-task and multi-task settings, plus strong-to-weak distillation scenarios, indicates broad applicability beyond specific use cases.

Future developments likely involve applying similar adaptive-window logic to other expensive training procedures, exploring whether prefix-level alignment patterns generalize across model architectures and domains, and investigating how this approach interacts with emerging inference-time scaling methods.

• →ADWIN reduces on-policy distillation training costs by up to 4.1x through adaptive rollout length decisions based on teacher-student alignment.
• →The framework uses short teacher-anchored prefixes for training while periodically conducting full-rollout audits to maintain alignment quality.
• →Performance remains comparable or better than full-rollout baselines across math and code reasoning benchmarks in multiple training settings.
• →Adaptive windowing with staleness control prevents the strategy from degrading as student models evolve during training.
• →The approach highlights that not all sequence positions contribute equally to learning, enabling significant compute-accuracy trade-off improvements.