Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

Researchers investigate on-policy distillation (OPD) dynamics in large language model training, identifying two critical success conditions: compatible thinking patterns between student and teacher models, and genuine new capabilities from the teacher. The study reveals that successful OPD relies on token-level alignment and proposes recovery strategies for failing distillation scenarios.

On-policy distillation has emerged as a foundational post-training technique for large language models, yet the mechanisms driving its success or failure remained largely opaque until this research. This paper advances the field by empirically characterizing what separates effective distillation from failed attempts, moving beyond trial-and-error approaches that plague model optimization workflows.

The research establishes two gatekeeping conditions for OPD success. First, student and teacher models must operate with compatible reasoning patterns—a finding validated through reverse distillation experiments showing that same-family models (1.5B and 7B variants) present indistinguishable distributions to the student. Second, teachers must provide genuinely novel capabilities rather than simply higher scores on existing training distributions. This distinction matters because many practitioners assume higher performance automatically transfers value, when in reality the student may have already seen similar patterns.

At the mechanistic level, the authors identify a concentrated token probability set (97-99% of mass) that governs successful distillation, with progressive alignment occurring primarily on high-probability tokens at student-visited states. This finding enables targeted interventions. The proposed recovery strategies—off-policy cold start and teacher-aligned prompt selection—offer practical tools for practitioners encountering distillation failures in production settings.

A significant implication emerges around scalability: the apparent "free lunch" of dense token-level rewards comes with hidden costs that may prevent scaling to long-horizon distillation tasks. This raises questions about OPD's effectiveness for increasingly complex reasoning requirements in frontier models, potentially motivating alternative post-training architectures that don't rely on dense token supervision.

• →Successful on-policy distillation requires both compatible thinking patterns and genuinely new teacher capabilities beyond student training data
• →Token-level alignment concentrates on a small shared probability set (97-99%), enabling targeted optimization strategies
• →Weak-to-strong reverse distillation reveals that same-family models may be distributionally indistinguishable from student perspectives
• →Off-policy cold start and teacher-aligned prompt selection provide practical recovery mechanisms for failing distillation scenarios
• →OPD's scalability to long-horizon tasks remains uncertain due to hidden costs in dense token-level reward structures