Data-Efficient On-Policy Distillation for Automatic Speech Recognition

Researchers demonstrate that a 0.6B-parameter ASR model trained on 100k hours of speech can achieve competitive performance with larger models through teacher-guided on-policy distillation, reducing the audio data requirements by 99.5% compared to industry standards while closing the capability gap with 1.7B parameter models.

This research addresses a critical bottleneck in automatic speech recognition: the massive computational and data collection costs required to build competitive models. The Ark-ASR project achieves meaningful performance improvements on Mandarin and English benchmarks using dramatically reduced training data—100k hours versus the 20 million hours consumed by comparable state-of-the-art systems. This efficiency breakthrough matters because it democratizes ASR model development, enabling smaller organizations and researchers to build specialized speech systems without prohibitive resource expenditures.

The technical approach leverages on-policy distillation, where a larger teacher model (Qwen-ASR) guides a smaller student model's learning process. Unlike traditional knowledge distillation that can suffer from distribution mismatches, on-policy methods keep the student's training distribution aligned with the teacher's expertise. The researchers validate this with a support-overlap diagnostic, confirming that teacher-data staging improves student-teacher compatibility—a finding that strengthens the theoretical foundation for this training methodology.

For the AI development ecosystem, this work signals that model efficiency gains may come not just from architectural innovations but from smarter training recipes. Organizations building production ASR systems face lower barriers to entry and can allocate resources toward domain specialization rather than raw data collection. The 200x reduction in audio hours required while maintaining competitive accuracy could accelerate deployment of ASR in underserved languages and specialized domains where large datasets remain unavailable. Future work should focus on whether these efficiency gains generalize across additional languages and whether further compression is possible with even smaller student models.

• →On-policy distillation reduces ASR training data requirements from 20M to 100k hours while maintaining competitive performance across multiple benchmarks.
• →A 0.6B-parameter model can achieve near-parity with larger baseline systems through strategic teacher-guided training rather than additional scale.
• →Support-overlap diagnostics validate that teacher-data staging improves compatibility between student and teacher models in on-policy distillation.
• →The 200x reduction in audio data opens accessibility for ASR development in resource-constrained settings and specialized domains.
• →Results demonstrate data-efficient training can substantially close capability gaps between compact and large-scale ASR models.