Where Reliability Lives in Vision-Language Models: A Mechanistic Study of Attention, Hidden States, and Causal Circuits

Researchers challenge the widespread assumption that sharp attention maps in vision-language models indicate reliable outputs. Through mechanistic analysis of three VLM families (LLaVA, PaliGemma, Qwen2-VL), they find attention structure is nearly uncorrelated with correctness, while hidden-state geometry and late-layer circuits prove far more predictive of model reliability.

The study directly interrogates a foundational intuition in vision-language model interpretability: that concentrated attention patterns correlate with confident, accurate predictions. Testing this assumption across 3,090 samples reveals attention concentration has virtually zero correlation with correctness (R=0.001), despite remaining mechanistically necessary for feature extraction. This disconnect between interpretability and reliability has significant implications for how researchers and practitioners should monitor and trust VLM outputs.

The findings emerge from a unified mechanistic pipeline—the VLM Reliability Probe—that instruments three major open-weight model families to compare attention maps, generation dynamics, and hidden-state representations against ground-truth labels. The work extends beyond attention, identifying that reliability becomes legible in deeper computational layers: hidden-state linear probes achieve AUROC>0.95 on benchmark tasks, and self-consistency checks at K=10 iterations show the strongest behavioral predictive power, though at tenfold inference cost.

Critically, the research uncovers an architectural split with design consequences. Late-fusion models like LLaVA concentrate reliability in fragile late-layer bottlenecks, where ablating just five neurons causes 8.3pp accuracy drops. Early-fusion variants (PaliGemma, Qwen2-VL) distribute reliability across layers, maintaining near-complete robustness even when 50% of peak-layer dimensions are destroyed. These architectural differences suggest fundamentally different failure modes and reliability profiles across model families.

• →Attention-map sharpness is a near-zero predictor of vision-language model correctness, contradicting common interpretability assumptions
• →Hidden-state geometry and sparse late-layer circuits prove far more reliable indicators of model accuracy than attention structure
• →Late-fusion architectures concentrate reliability in brittle bottlenecks while early-fusion designs distribute it robustly across layers
• →Self-consistency verification at K=10 is the strongest behavioral reliability predictor measured, though requiring 10x inference overhead
• →Mechanistic analysis reveals distinct architectural vulnerabilities requiring different monitoring and safety strategies per model family