OPRD: On-Policy Representation Distillation

Researchers propose On-Policy Representation Distillation (OPRD), a novel method for training smaller AI models by aligning hidden-state representations with teacher models rather than just matching output probabilities. OPRD achieves superior performance on mathematical reasoning benchmarks while training 1.44x faster and using 54% less memory than existing approaches.

OPRD addresses fundamental limitations in current model distillation techniques by shifting supervision from output space to intermediate representation space. Traditional on-policy distillation methods rely on Monte Carlo sampling to estimate KL divergence across large vocabularies—a process that introduces persistent variance, particularly problematic for models like Qwen with ~150k tokens. By operating directly on hidden states, OPRD eliminates this sampling noise while extracting richer structural information from each layer of the teacher model.

The research builds on longstanding challenges in knowledge distillation for large language models. As models scale to hundreds of billions of parameters, the computational and memory costs of deploying them become prohibitive. Previous distillation approaches treated teacher models as black boxes, discarding valuable intermediate representations. OPRD's innovation lies in leveraging these hidden states during training on identical rollouts, creating a more direct learning signal for student networks.

The empirical results demonstrate practical significance: OPRD closes performance gaps on AIME 2024/2025 and AIMO benchmarks where output-only baselines stagnate, suggesting the method captures mathematical reasoning patterns more effectively. The 1.44x training speedup and 54% memory reduction directly impact deployment economics, making smaller models more practical for resource-constrained environments.

This development matters for the broader AI infrastructure industry as it enables more efficient model compression without sacrificing capability. Organizations can deploy smaller, faster models without the performance degradation typical of existing distillation methods, reducing inference costs and energy consumption while maintaining reasoning quality.

• →OPRD eliminates sampling variance by operating in hidden-state space rather than output probability space
• →Method achieves superior performance on mathematical reasoning benchmarks compared to output-only distillation baselines
• →Training efficiency improves by 1.44x with 54% less memory consumption than top-k on-policy distillation
• →Approach leverages intermediate layer representations that traditional black-box distillation discards
• →Results suggest potential for more practical deployment of high-capability models in resource-constrained settings