TARAC: Mitigating Hallucination in LVLMs via Temporal Attention Real-time Accumulative Connection

Researchers introduce TARAC, a training-free framework that mitigates hallucinations in Large Vision-Language Models by dynamically preserving visual attention across generation steps. The method achieves significant improvements—reducing hallucinated content by 25.2% and boosting perception scores by 10.65—while adding only ~4% computational overhead, making it practical for real-world deployment.

Large Vision-Language Models (LVLMs) have become increasingly capable but remain prone to hallucinations—generating plausible-sounding but factually incorrect descriptions of images. This limitation stems from attention decay during the generation process, where models gradually lose visual grounding as they produce longer sequences. TARAC addresses this core problem through a novel mechanism that accumulates and re-injects historical attention weights in real-time, mimicking cognitive reinforcement processes without requiring model retraining.

The hallucination problem in LVLMs has attracted significant research attention as these models see wider deployment in applications demanding accuracy. Previous solutions either impose substantial computational costs or require expensive retraining procedures that limit their practical applicability. TARAC's training-free approach positions it as a genuinely practical solution that can be retrofitted onto existing models.

The framework demonstrates impressive empirical results across multiple architectures including LLaVA and Qwen2-VL, validated on established benchmarks like CHAIR and MME. The negligible inference overhead—approximately 4% increase in time-per-output-token—contrasts sharply with the double-digit percentage increases typical of competing training-free methods. This efficiency-effectiveness tradeoff directly benefits developers deploying these systems in production environments where latency constraints are critical.

The work signals maturation in the LVLM hallucination mitigation space, shifting from research-focused solutions toward practically deployable fixes. As vision-language models integrate into commercial applications across content creation, accessibility, and autonomous systems, solutions like TARAC become essential infrastructure rather than optional improvements.

• →TARAC reduces hallucinated sentences by 25.2% on CHAIR benchmarks while maintaining near-imperceptible inference overhead
• →The framework operates as a plug-and-play training-free module compatible with existing LVLMs without requiring retraining
• →Historical attention accumulation mechanism draws inspiration from cognitive reinforcement, sustaining visual grounding during generation
• →Results validated across multiple architectures (LLaVA, Qwen2-VL) demonstrate broad applicability beyond single-model implementations
• →Efficiency gains enable practical deployment where previous training-free hallucination mitigation methods imposed prohibitive computational costs