Deeper Thought, Weaker Aim: Understanding and Mitigating Perceptual Impairment during Reasoning in Multimodal Large Language Models

Researchers have identified that multimodal large language models (MLLMs) lose visual focus during complex reasoning tasks, with attention becoming scattered across images rather than staying on relevant regions. They propose a training-free Visual Region-Guided Attention (VRGA) framework that improves visual grounding and reasoning accuracy by reweighting attention to question-relevant areas.

• →MLLMs suffer from attention dispersion during multi-step reasoning, causing them to lose focus on visually relevant regions.
• →Extended reasoning prompts significantly reduce the model's attention to image areas critical for answering questions.
• →There is a strong correlation between overall attention on image tokens and spatial dispersiveness within images.
• →The proposed VRGA framework requires no additional training and uses entropy-focus criteria to select and reweight visual attention heads.
• →Experimental results show the method effectively improves visual grounding and reasoning accuracy while providing interpretable insights.