Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought

Researchers propose SFFL, a framework that mitigates cross-modal interference in audio-visual language models by enforcing separate reasoning chains for each modality before fusion. The approach uses modality-preference labels and reinforcement learning to reduce hallucinations and achieves 5-11% performance improvements on benchmarks.

The research addresses a fundamental challenge in multimodal AI: when models process audio and visual information simultaneously, they often experience interference where one modality misguides interpretation of another, leading to hallucinations and inaccurate outputs. This problem becomes increasingly critical as multimodal models expand into real-world applications requiring reliable audio-visual reasoning, from surveillance systems to accessibility tools.

The SFFL framework represents a methodological shift in how multimodal models handle information fusion. Rather than allowing cross-modal interaction throughout reasoning, the approach isolates each modality's reasoning chain initially, then selectively integrates evidence. The researchers construct modality-preference labels through systematic data pipelines and use reinforcement learning to teach models when to rely on specific modalities. This architectural choice reflects growing recognition that uncontrolled information mixing in neural networks amplifies rather than resolves ambiguity.

For AI developers, this work directly impacts model reliability and robustness. The 11.17% improvement on cross-modal hallucination benchmarks suggests SFFL meaningfully addresses a practical limitation affecting deployed systems. The framework's instance-dependent preference mechanism ensures adaptability across diverse scenarios rather than applying blanket modality weightings.

Looking forward, this research likely influences how multimodal LLMs incorporate new modalities beyond audio-vision pairs. The modality-specific chain-of-thought pattern provides a template for managing complexity as models integrate more information streams. Subsequent research may explore whether similar separation-then-fusion principles extend to text-audio-visual models or other multimodal combinations.

• →SFFL framework reduces cross-modal interference by enforcing separate reasoning chains for audio and visual modalities before evidence fusion
• →Modality-preference labels and reinforcement learning enable instance-dependent weighting of audio versus visual cues
• →Approach achieves 5.16% average improvement on general AVQA benchmarks and 11.17% on cross-modal hallucination tests
• →Modality isolation during reasoning phase combined with full cross-modal access during fusion stage optimizes information usage
• →Research addresses hallucination problems affecting real-world multimodal AI applications