Multi-Teacher Knowledge Distillation via Teacher-Informed Mixture Priors

Researchers introduce Multi-Teacher Bayesian Knowledge Distillation (MT-BKD), a framework that enables student models to learn from multiple teacher models while quantifying uncertainty through Bayesian inference. The approach uses teacher-informed priors and entropy-based weighting to improve model compression, generalization, and interpretability across synthetic and real-world tasks.

Knowledge distillation has emerged as a critical technique for deploying large models efficiently, but the field has lacked rigorous statistical frameworks for understanding how students learn from teachers and managing uncertainty in real-world applications. MT-BKD addresses this gap by integrating Bayesian inference into multi-teacher distillation, fundamentally shifting how practitioners approach model compression. Rather than treating distillation as a deterministic process, the framework captures inherent uncertainty, providing confidence estimates alongside predictions—a feature increasingly demanded in high-stakes applications like healthcare and autonomous systems.

The technical innovation centers on teacher-informed priors that incorporate external knowledge from multiple educators alongside task-specific training data. This approach enables more robust generalization because the student model learns not just to mimic teacher outputs, but to understand the uncertainty landscape across different expert perspectives. The entropy-based weighting mechanism represents an adaptive approach to the classical problem of combining expert opinions, automatically calibrating how much influence each teacher exerts during training.

For the AI development community, this work matters because uncertainty quantification has become non-negotiable for deployment in regulated industries. The protein subcellular location and image classification experiments demonstrate practical utility, but the real value emerges in domains requiring model confidence estimates. This framework could accelerate adoption of distilled models in production environments where practitioners previously hesitated due to calibration concerns. Organizations building compressed models for edge deployment will find the uncertainty estimates particularly valuable for detecting out-of-distribution inputs and triggering model retraining.

• →MT-BKD integrates Bayesian inference into knowledge distillation to quantify uncertainty alongside model predictions
• →Teacher-informed priors incorporate multiple expert perspectives, improving generalization and robustness compared to single-teacher approaches
• →Entropy-based weighting automatically adjusts each teacher's influence, adapting to varying levels of expertise across domains
• →Framework enhances interpretability of student learning while maintaining computational efficiency benefits of model compression
• →Validation on protein prediction and image classification shows practical improvements in both accuracy and uncertainty calibration