Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation

Researchers developed Token-Selective Dual Knowledge Distillation (TSD-KD), a new framework that improves AI reasoning by allowing smaller models to learn from larger ones more effectively. The method achieved up to 54.4% better accuracy than baseline models on reasoning benchmarks, with student models sometimes outperforming their teachers by up to 20.3%.

• →TSD-KD framework enables more efficient knowledge transfer from large AI models to smaller ones for reasoning tasks.
• →The method uses both indirect feedback through preference ranking and selective token distillation to avoid overwhelming smaller models.
• →Student models trained with TSD-KD outperformed baseline methods by up to 54.4% on challenging reasoning benchmarks.
• →In four cases, student models actually exceeded their teacher model performance by up to 20.3%.
• →The approach allows smaller models to develop reasoning in their own words rather than mimicking entire teacher distributions.