Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders

Researchers demonstrate that Whisper, OpenAI's widely-used speech recognition model, can detect and mitigate hallucinations—fabricated coherent transcriptions from non-speech audio—using Sparse AutoEncoders and activation-space steering. The approach reduces hallucination rates from 72-87% to 14-27% across model sizes with minimal performance degradation on actual speech.

Whisper's hallucination problem represents a critical vulnerability in production ASR systems. When fed silence, music, or noise, the model generates plausible but entirely fabricated transcriptions, creating reliability issues for applications from transcription services to accessibility tools. This research tackles the problem at the representation level rather than through expensive retraining, offering a practical mitigation pathway for deployed systems.

The key insight is that hallucination patterns are detectable in Whisper's internal activations before they propagate to outputs. By analyzing both raw encoder activations and Sparse AutoEncoder (SAE) latents, the researchers found that hallucination-related features concentrate in sparse subsets and strengthen in deeper layers. This suggests the model encodes distinguishable patterns between genuine speech processing and hallucination generation—patterns exploitable for steering.

The practical impact favors developers maintaining Whisper deployments. SAE-based steering achieves near fine-tuning performance without model modification, addressing a known limitation affecting production quality. The 58-point reduction in hallucination rate on non-speech inputs (72% to 14% for Whisper small) substantially improves safety for applications where false transcriptions carry real consequences—medical dictation, legal proceedings, or accessibility features.

Future implications extend beyond Whisper to broader LLM and multimodal systems exhibiting similar hallucination pathologies. The methodology demonstrates that steering through sparse feature identification offers scalable mitigation compared to resource-intensive fine-tuning. Monitoring whether similar techniques transfer to other models and modalities will indicate whether this represents a generalizable approach to hallucination control.

• →Whisper hallucinations are detectable and correctable through internal representation steering without model retraining
• →Sparse AutoEncoder latent-space steering reduces hallucination rates by 58-60 percentage points on non-speech audio
• →Hallucination-related features concentrate in sparse subsets and intensify in deeper encoder layers
• →The method maintains low performance degradation on legitimate speech data, approaching fine-tuning baselines
• →This representation-based approach scales across model sizes and may generalize to other multimodal systems