ASKD-Whisper: Adaptive Self-knowledge Distillation for Efficient and Low-Latency Automatic Speech Recognition

Researchers propose ASKD-Whisper, a new knowledge distillation technique that compresses OpenAI's Whisper speech recognition model while improving performance. The method achieves 5x faster inference and 1.07% lower error rates than the original teacher model by dynamically reducing reliance on the teacher's predictions during training.

Knowledge distillation—the process of compressing large AI models into smaller, deployable versions—traditionally forces student models to mimic teacher predictions exactly. This approach accelerates learning but introduces a critical vulnerability: students inherit the teacher's blind spots and overconfident errors, especially on data outside the training domain. The ASKD framework addresses this through a dynamic curriculum that systematically decreases teacher dependency as training progresses, then applies self-distillation as a regularization mechanism. This allows the student model to develop independent reasoning capacity while maintaining stability.

The breakthrough centers on preventing what researchers call "teacher-induced overfitting." While previous distillation methods achieved compression through mimicry, ASKD-Whisper demonstrates that selective independence during training produces superior generalization. The results are impressive: a 5x speedup in inference latency combined with measurable accuracy improvements suggests the approach fundamentally rethinks the student-teacher dynamic.

For the AI industry, this research has significant practical implications. Efficient speech recognition models enable deployment on edge devices, mobile applications, and resource-constrained environments—markets currently dominated by cloud-based solutions. Whisper's multilingual capabilities combined with improved generalization make ASKD-Whisper attractive for enterprises seeking both performance and cost reduction. The technique generalizes beyond speech recognition to other large foundation models, potentially influencing how companies compress language models, vision transformers, and multimodal architectures.

The research validates an emerging principle: better compression comes not from stricter teacher alignment but from strategic autonomy during training. Future work likely explores applying adaptive distillation to other domains and scaling to even larger teacher models, establishing new efficiency benchmarks across AI applications.

• →ASKD-Whisper achieves 5x inference speedup while reducing word error rates by 1.07% compared to the original Whisper model.
• →Dynamic curriculum learning that decays teacher dependency prevents student models from inheriting teacher blind spots and hallucinations.
• →The technique enables efficient speech recognition deployment on edge devices and mobile platforms with superior out-of-distribution generalization.
• →Adaptive self-distillation represents a paradigm shift from static mimicry-based compression toward dynamic, independence-enabling training protocols.
• →The framework has potential applications across foundation model compression beyond speech recognition, including language and vision models.