Late-Layer Fusion is Enough: Dual-Path Vision Token Routing for Multimodal Large Language Models under Visual Saturation

Researchers propose Dual-Path Vision Token Routing (DPVR), a framework that optimizes multimodal large language models by routing vision tokens away from deep transformer layers where they saturate early, instead fusing visual and textual information only in the final layer. The approach reduces computational overhead by 3% while maintaining competitive performance, challenging the assumption that vision tokens must traverse all deep language-model layers.

This research addresses a fundamental architectural inefficiency in multimodal large language models that has gone largely unexamined. Current MLLMs like LLaVA-1.5 apply symmetric transformer processing to both image and text tokens despite their substantially different characteristics—vision tokens saturate in information around middle layers while text tokens continue benefiting from deeper processing. The analysis revealing text-to-image attention dropping from 0.68 at layer 0 to 0.07 by layer 4 demonstrates quantifiable evidence of this asymmetry.

The proposed DPVR-LF solution is elegantly simple: route vision tokens into a trainable side branch after saturation, skip visual positions in thirteen deep transformer layers, and reunite modalities at the final layer. This approach reduces redundant visual computation while preserving model performance on standard benchmarks. The framework's efficiency gains—achieved with just 3% additional trainable parameters—suggest substantial room for optimization in current MLLM architectures.

For the AI development community, this work has immediate implications for model efficiency and cost reduction. As multimodal models become increasingly deployed, eliminating unnecessary visual computation could significantly decrease inference latency and computational requirements without sacrificing capability. This is particularly valuable for edge deployment and real-time applications.

The findings invite architectural rethinking across the MLLM ecosystem. Developers may now reconsider symmetric transformer designs, potentially exploring purpose-built pathways for different modalities. Future research will likely extend these concepts to other modality combinations and examine whether similar saturation patterns exist in other multimodal architectures.

• →Vision tokens in MLLMs saturate in middle layers while text tokens benefit from deeper processing, indicating architectural asymmetry
• →Late-layer fusion of visual and textual streams maintains performance while reducing computational overhead by eliminating redundant deep visual processing
• →DPVR-LF achieves efficiency gains with only 3% additional trainable parameters, suggesting broader optimization opportunities in current model designs
• →The research challenges the conventional assumption that vision tokens must traverse all transformer layers in multimodal models
• →Findings have direct implications for reducing inference latency and computational costs in deployed multimodal AI systems