From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs

Researchers have mapped how Audio-Visual Large Language Models (AVLLMs) process and integrate audio and visual information internally, revealing distinct information flow patterns depending on input configuration. The study demonstrates that multimodal tokens can be pruned after information transfer with minimal performance impact, enabling more efficient inference across different model scales.

Understanding how multimodal language models process sensory information represents a critical frontier in AI interpretability. This research provides the first systematic analysis of information flow within AVLLMs, moving beyond treating these models as black boxes to reveal their internal decision-making architecture. The findings demonstrate that AVLLMs adapt their routing strategies based on input structure—following sequential pathways for video content while switching to parallel streams for interleaved multimodal items. This adaptive behavior suggests models learn task-specific modality weighting, allocating processing resources proportionally to whether audio or visual information drives the final prediction.

The efficiency implications extend beyond academic interest. By proving that audio-visual tokens become redundant after their information transfers to the language model core, researchers have identified concrete optimization opportunities. Token pruning in AVLLMs mirrors similar compression techniques in vision transformers but with added complexity from multiple modalities. This finding holds across different architectures (Qwen2.5-Omni and Video-SALMONN2) and scales (3B and 7B parameters), suggesting the principle generalizes rather than exploiting specific model quirks.

For the broader AI ecosystem, these insights influence both deployment efficiency and future architecture design. Practitioners can reduce computational overhead during inference while maintaining or slightly improving accuracy. Model developers gain theoretical understanding of why certain connection patterns emerge, informing next-generation designs. As AVLLMs proliferate in edge computing and real-time applications, such efficiency gains become practically valuable. The research establishes interpretability methodology applicable to increasingly complex multimodal systems, setting precedent for transparency in AI development.

• →AVLLMs route information sequentially for unified audio-visual video but switch to parallel streams for multiple interleaved items, demonstrating adaptive processing strategies.
• →Audio and visual tokens can be discarded after their information transfers to the language model without degrading performance, enabling more efficient inference.
• →The findings generalize across multiple model architectures and scales, suggesting fundamental principles underlie multimodal information flow.
• →Task-specific modality contribution aligns with the sequential information pathway, with each modality contributing proportionally to its relevance.
• →This interpretability research provides a foundation for designing more efficient and transparent audio-visual and broader multimodal language models.