Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation

Researchers identify Supervision Fidelity Decay (SFD) as a critical limitation in on-policy distillation where teacher model confidence deteriorates as student-generated reasoning chains lengthen. They propose Lookahead Group Reward (LGR) with entropy-triggered tree-attention to strengthen supervision signals, achieving 2.57-point improvements on math and code benchmarks, with gains reaching 4.92 points on AIME-26.

On-policy distillation represents a sophisticated approach to model training where a smaller student model learns from a larger teacher by generating its own trajectories and receiving token-level feedback. The research identifies a previously underexplored degradation mechanism: as student-generated prefixes grow longer, the teacher model's ability to provide confident and discriminative guidance deteriorates significantly. This phenomenon compounds student errors across extended reasoning chains, particularly problematic for complex mathematical and coding tasks requiring multi-step logic.

The proposed Lookahead Group Reward mechanism addresses this through a novel insight: future teacher confidence at the next step can predict the strength of corrective supervision. By evaluating candidate tokens based on the confidence they induce downstream rather than immediate predictions, the approach preserves signal fidelity across longer sequences. The entropy-triggered tree-attention mechanism ensures computational efficiency, making the solution practical for deployment.

This work carries implications for AI model development efficiency. Stronger on-policy distillation directly enables more capable smaller models, reducing computational requirements while maintaining reasoning quality. The 4.92-point improvement on AIME-26 at 39k tokens demonstrates particular relevance for code and mathematics domains where extended chain-of-thought reasoning dominates.

For the broader machine learning community, this research opens pathways to improving student model training without architectural changes or additional data. Future work likely explores whether LGR principles apply to other distillation paradigms or scaling scenarios, potentially reshaping how organizations approach model efficiency and capability tradeoffs.

• →Supervision Fidelity Decay degrades teacher guidance quality as student-generated sequences lengthen, compounding errors in reasoning tasks
• →Lookahead Group Reward mechanism improves distillation by evaluating tokens based on downstream teacher confidence rather than immediate predictions
• →Results show 2.57-point mean improvement across six benchmarks, with 4.92-point gains on AIME-26 long-generation tasks
• →Entropy-triggered tree-attention reduces computational overhead while maintaining performance benefits across extended token sequences
• →Approach enables more efficient model distillation without requiring architectural changes or additional training data