Acoustic Cue Alignment in Audio Language Models for Speech Emotion Recognition

Researchers demonstrate that instruction-following audio language models can effectively utilize explicit acoustic cues for speech emotion recognition, with aligned acoustic tokens improving performance on standard benchmarks while remaining grounded in the underlying audio signal.

This research addresses a fundamental question about how modern audio language models integrate symbolic representations with raw acoustic data. The study employs a methodical approach by deriving interpretable acoustic concept tokens from the eGeMAPS feature set—a standardized paralinguistic framework—and appending them to text prompts while preserving the original audio input. Testing on FAU-Aibo and IEMOCAP benchmarks reveals that aligned tokens enhance unweighted average recall, while corrupted or conflicting tokens degrade performance, indicating the models genuinely process these symbolic cues rather than ignoring them.

The robustness findings prove particularly significant: models maintain performance despite strong token perturbations, suggesting they maintain grounding in the raw audio signal even when symbolic channels provide conflicting information. This partial anchoring to audio represents a more sophisticated behavior than simple token-following would produce. The research contributes to understanding multimodal integration in language models, where combining multiple information streams creates more reliable representations than either modality alone.

For the broader AI research community, this work establishes token-based interventions as a practical probing method for interpretability in audio-grounded systems. This has implications for affective computing applications, where understanding model behavior proves critical for deployment in sensitive contexts like mental health monitoring or user experience optimization. The methodology itself—systematically perturbing symbolic inputs while measuring acoustic grounding—offers transferable insights for other multimodal AI systems beyond speech emotion recognition, enabling researchers to dissect how language models reconcile competing information sources.

• →Aligned acoustic concept tokens improve speech emotion recognition performance on standard benchmarks while maintaining grounding in raw audio.
• →Models remain partly anchored to audio signals even under strong token perturbations, indicating robust multimodal integration rather than simple token-following behavior.
• →Token-based interventions provide a practical method for probing interpretability and robustness in audio language models used for affective computing.
• →Conflicting or corrupted tokens shift model confusion patterns toward neutral predictions, revealing how symbolic cues influence decision boundaries.
• →The methodology demonstrates transferable insights for understanding multimodal information reconciliation in other language model applications.