Bangla-WhisperDiar: Fine-Tuning Whisper and PyAnnote for Bangla Long-Form Speech Recognition and Speaker Diarization

Researchers have developed Bangla-WhisperDiar, a fine-tuned speech recognition and speaker diarization system that achieves a 24.41% word error rate for ASR and 23.92% diarization error rate. The work addresses critical gaps in Bangla language processing by combining OpenAI's Whisper model with PyAnnote's diarization framework, trained on custom datasets with extensive data augmentation techniques.

This research tackles a significant underserved problem in natural language processing: accurate speech recognition and speaker identification for Bangla, a language spoken by over 300 million people. The technical achievement demonstrates meaningful progress in handling long-form audio processing, which remains computationally challenging and acoustically complex. The 24.41% WER and 23.92% DER represent substantial improvements over baseline models, indicating that targeted fine-tuning on curated datasets can effectively address language-specific challenges.

The broader context reveals a growing focus on expanding AI capabilities beyond English-dominant systems. Most commercial speech recognition tools historically underperform on low-resource and regional languages, creating accessibility gaps for non-English speakers. This work follows the industry trend of democratizing advanced NLP capabilities through open-source models like Whisper and PyAnnote, which serve as strong foundation models requiring relatively modest computational resources for fine-tuning.

For the AI and speech technology sectors, this research signals market demand for multilingual ASR solutions. Organizations developing applications in South Asian markets—customer service platforms, accessibility tools, content moderation systems—now have validated methodologies for deploying production-grade Bangla speech systems. The detailed documentation of data augmentation strategies and training approaches provides a replicable framework for other low-resource language communities.

Looking ahead, the critical challenge involves scaling this approach to other underrepresented languages while addressing real-world deployment constraints like latency and computational efficiency. Success here could accelerate similar projects across African, Southeast Asian, and indigenous language communities, fundamentally expanding who can benefit from AI-powered speech technologies.

• →Fine-tuned Whisper and PyAnnote models achieve 24.41% WER and 23.92% DER on Bangla speech tasks, significantly improving baseline performance
• →Comprehensive data augmentation techniques including noise injection, reverb, and pitch perturbation proved essential for handling diverse acoustic conditions
• →The work demonstrates that targeted training on 15,000 curated Bangla audio segments enables production-viable speech recognition systems
• →Open-source foundation models enable rapid development of specialized ASR solutions for low-resource language communities
• →Detailed methodology documentation provides a replicable framework for advancing speech technology in other underrepresented languages