Beyond the Mouth: Upper-Face Affective Cues in Audiovisual Sentence Recognition under Acoustic Uncertainty

A new study demonstrates that upper-face affective cues significantly enhance audiovisual speech recognition systems when audio quality degrades, particularly in noisy environments. Rather than encoding linguistic content directly, emotional facial expressions improve model calibration and robustness, suggesting that human communication relies on socially expressive signals beyond traditional mouth-region visual cues.

This research addresses a fundamental gap in multimodal AI systems by examining how affective facial information complements acoustic and articulatory cues during speech comprehension. Traditional audiovisual speech recognition systems prioritize mouth-region features as the primary visual linguistic source, treating facial expressions as separate emotion-recognition targets. The study uses the CREMA-D corpus to systematically evaluate how upper-face features contribute to sentence recognition accuracy across varying noise levels, revealing nuanced insights about human-centered multimodal interaction.

The findings indicate that mouth and lower-face features provide substantial robustness improvements under acoustic degradation, with accuracy gains of nearly 8% at 0 dB SNR. Upper-face affective cues show a more subtle pattern: while direct accuracy improvements from adding upper-face information to audio-plus-mouth models are modest, full-face models consistently enhance calibration metrics and outperform controls with shuffled upper-face data under noisy conditions. This suggests affective expressions serve a different functional role than articulatory cues, potentially signaling speaker confidence or emotional state rather than phonetic content.

For developers building practical audiovisual systems, this research validates incorporating full-face analysis beyond mouth-centric approaches, particularly for applications prioritizing confidence estimation and user experience in degraded acoustic environments. Speech-recognition systems deployed in noisy settings—call centers, public spaces, assistive technologies—could benefit from leveraging upper-face information for improved calibration and user trust. The work emphasizes that socially expressive cues enhance system robustness through mechanisms distinct from traditional linguistic signal processing, opening opportunities for more human-aligned AI design that acknowledges communication's inherently social dimensions.

• →Upper-face affective cues improve model calibration and robustness in noisy audio conditions without directly encoding lexical content
• →Mouth-region features provide 7.94% accuracy improvement over audio-only systems at 0 dB SNR, demonstrating substantial robustness benefits
• →Full-face models outperform audio-plus-mouth baselines in confidence estimation across all signal-to-noise ratios tested
• →Affective facial expressions function differently from articulatory cues, suggesting multiple complementary channels in multimodal speech comprehension
• →Findings support broader integration of socially expressive features in audiovisual AI systems for improved human-centered interaction