TRACE: Distilling Where It Matters via Token-Routed Self On-Policy Alignment

Researchers introduce TRACE, a novel training method that improves AI model performance by selectively applying different optimization techniques to critical versus routine tokens in reasoning tasks. The approach addresses inefficiencies in standard self-distillation by concentrating training effort on important decision points, achieving 2.76 percentage point improvements over baseline methods while better preserving out-of-distribution generalization.

TRACE represents a meaningful refinement in how large language models learn from self-generated feedback during reinforcement learning training. The core innovation addresses a fundamental inefficiency: when models teach themselves using privileged information (annotations from human experts), applying identical learning pressure across all tokens wastes computational resources and introduces subtle degradation in reasoning quality. By routing different optimization strategies to specific token regions—forward KL divergence for critical reasoning steps, reverse KL for error correction, and standard RL for routine tokens—the method achieves better resource allocation.

This work emerges from growing recognition that not all tokens contribute equally to model reasoning quality. Standard approaches treat language generation as uniform, but human annotations typically mark only specific spans as critical for correctness. TRACE operationalizes this insight through selective gradient application, preventing the entropy collapse and distribution drift observed in competing methods. The scalability analysis revealing that optimal strategies vary by model size (8B versus 1.7B) suggests token routing effectiveness depends on model capacity, a nuanced finding for practitioners.

The implications extend beyond pure performance metrics. TRACE's ability to maintain out-of-distribution performance on GPQA-Diamond while improving on in-distribution benchmarks directly addresses a persistent challenge in scaling AI systems: maintaining robustness as models optimize for specific tasks. The demonstration that online self-annotation recovers 69% of strong external annotation gains proves the method doesn't merely import annotator capability but genuinely improves model learning efficiency. For AI development, this signals progress toward training methods that scale more intelligently rather than simply increasing compute allocation.

• →TRACE improves math reasoning performance by 2.76 percentage points through selective token-level optimization routing
• →The method preserves out-of-distribution generalization where standard baselines degrade, addressing a critical scalability concern
• →Token-specific learning strategies (forward KL, reverse KL, or RL) vary optimally by model size, indicating capacity-dependent training dynamics
• →Self-annotation yields 69% of external annotation gains, demonstrating genuine learning improvements rather than capability borrowing
• →Selective gradient application prevents entropy collapse and information leakage inherent in full-token distillation approaches