HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation

Researchers introduce Hybrid Distillation Policy Optimization (HDPO), a new method that improves large language model training for mathematical reasoning by addressing 'cliff prompts' where standard reinforcement learning fails. The technique uses privileged self-distillation to provide learning signals for previously unsolvable problems, showing measurable improvements in coverage metrics while maintaining accuracy.

• →HDPO addresses a fundamental problem in RL training where models cannot learn from problems they completely fail to solve.
• →The method uses privileged self-distillation where the same model acts as both teacher and student with different inputs.
• →Experiments show consistent improvements in pass rates (+0.8-1.1% for pass@4, +0.4-1.7% for pass@8) while maintaining greedy accuracy.
• →The approach provides provably bounded realizability gap unlike traditional cross-model distillation methods.
• →The technique offers direct control over exploration-exploitation tradeoffs through adjustable distillation weights.