When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models

Researchers identify a fundamental geometric flaw in decoder-based Vision-Language Models where visual embeddings become over-aligned with linguistic patterns, causing systematic hallucinations. The study introduces quantitative methods to characterize this bias and proposes training-free and fine-tuning solutions that reduce hallucinations across multiple benchmarks without computational overhead.

Vision-Language Models have become critical infrastructure for high-stakes applications, yet their tendency to hallucinate—confidently describing non-existent visual content—represents a significant reliability gap. This research moves beyond treating hallucinations as isolated failures and instead identifies a root geometric cause: the over-alignment of visual embeddings with text manifolds creates a statistical linguistic bias that systematically dominates over fine-grained visual information. The mechanistic approach here is notable because it traces the problem to universal, dataset-agnostic properties of text subspaces, suggesting the issue is structural rather than data-dependent.

This work addresses a critical limitation in prior approaches. Previous solutions either aggressively closed the modality gap or relied on expensive black-box decoding strategies without addressing underlying mechanisms. By demonstrating that linguistic bias concentrates in top principal components of a text subspace, the researchers enable targeted interventions. The dual solution—a training-free inference method and a bias-aware fine-tuning paradigm—demonstrates flexibility in deployment contexts, with the inference variant offering particular value for practitioners unable to retrain models.

The practical implications extend across medical imaging, autonomous systems, and other high-stakes domains where hallucination errors carry significant costs. Improvements across POPE, CHAIR, and AMBER benchmarks plus stronger CLAIR scores on long-form captioning suggest broad applicability. For developers integrating VLMs into production systems, these techniques offer immediate gains without retraining costs. The research advances the field's ability to understand and mitigate failure modes in multimodal AI, establishing precedent for mechanistic approaches to model reliability rather than black-box mitigation strategies.

• →Vision-Language Models hallucinate due to geometric over-alignment between visual and text embeddings, not fundamental data limitations.
• →Linguistic bias concentrates predictably in top principal components of universal text subspaces, enabling targeted removal strategies.
• →Proposed training-free inference method reduces hallucinations with zero computational overhead compared to base models.
• →Solutions improve performance across multiple hallucination benchmarks (POPE, CHAIR, AMBER) and long-form captioning tasks.
• →Research enables practical deployment options for both resource-constrained and fine-tuning-capable practitioners.