From Oracle to Noisy Context: Mitigating Contextual Exposure Bias in Speech-LLMs

Researchers developed a new training framework to address contextual exposure bias in Speech-LLMs, where models trained on perfect conversation history perform poorly with error-prone real-world context. Their approach combines teacher error knowledge, context dropout, and direct preference optimization to improve robustness, achieving WER reductions from 5.59% to 5.17% on TED-LIUM 3.

• →Speech-LLMs suffer from contextual exposure bias when trained on perfect oracle history but deployed with error-prone real conversation context.
• →The proposed unified training framework uses Whisper hypotheses, context dropout, and Direct Preference Optimization to improve robustness.
• →Experiments showed consistent improvements on both in-domain TED-LIUM 3 and zero-shot LibriSpeech datasets under realistic conditions.
• →The DPO approach demonstrated best resilience against irrelevant-context attacks with minimal performance degradation.
• →Code and models are publicly available, enabling broader adoption of the contextual speech recognition improvements.