Focus Matters: Phase-Aware Suppression for Hallucination in Vision-Language Models

Researchers developed a new method to reduce hallucinations in Large Vision-Language Models (LVLMs) by identifying a three-phase attention structure in vision processing and selectively suppressing low-attention tokens during the focus phase. The training-free approach significantly reduces object hallucinations while maintaining caption quality with minimal inference latency impact.

• →Vision encoders in LVLMs follow a consistent three-phase structure: diffusion, focus, and rediffusion during visual information processing.
• →Hallucination behavior is particularly sensitive to tokens receiving low attention during the focus phase.
• →The proposed method operates training-free using statistics from a single forward pass and employs Determinantal Point Process to preserve visual diversity.
• →Experiments show consistent hallucination reduction across multiple LVLM backbones while maintaining competitive caption quality.
• →The approach achieves comparable hallucination mitigation to existing methods with negligible additional inference latency.