Self-Aug: Query and Entropy Adaptive Decoding for Large Vision-Language Models

Researchers developed a new training-free decoding strategy for Large Vision-Language Models that reduces hallucinations by using query-adaptive visual augmentation and entropy-based token selection. The method showed significant improvements in factual consistency across four LVLMs and seven benchmarks compared to existing approaches.

• →New decoding strategy addresses hallucination problems in Large Vision-Language Models without requiring additional training.
• →Self-augmentation prompting aligns semantics between text queries and visual augmentations using the model's intrinsic knowledge.
• →Adaptive thresholding algorithm adjusts token candidate selection based on output sparsity and logit distribution information.
• →Testing across four LVLMs and seven benchmarks demonstrated superior factual consistency compared to state-of-the-art methods.
• →The approach highlights the importance of query-dependent augmentation for improving LVLM generation quality.