Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbation

Researchers introduce Decoding by Perturbation (DeP), a training-free method that reduces hallucinations in multimodal large language models by applying controlled textual perturbations during decoding. The approach addresses the core issue where language priors override visual evidence, achieving improvements across multiple benchmarks without requiring model retraining or visual manipulation.

The hallucination problem in multimodal AI systems represents a critical bottleneck in deploying these models reliably. MLLMs often generate plausible-sounding but factually incorrect outputs because their language components dominate decision-making over actual visual signals. This DeP framework reframes hallucinations as an interpretability problem—revealing how textual phrasing influences visual grounding with excessive sensitivity. Rather than attacking the symptom through visual perturbations that distort image data, the method targets the root cause by interrogating the model's latent language priors through controlled textual variations.

The significance of this work lies in its training-free nature, eliminating expensive retraining cycles while maintaining model generative quality. By leveraging attention variance patterns and logits statistics, DeP constructs a quantifiable "prior drift direction" that counteracts probability biases arising from statistical co-occurrences in training data. This approach preserves the model's inherent fluency while selectively suppressing spurious confidence signals. The framework represents a shift toward mechanistic understanding of MLLM failure modes rather than brute-force mitigation strategies.

For practitioners deploying MLLMs in production environments, this development offers immediate value. Reducing hallucinations without fine-tuning enables faster deployment cycles and lower computational overhead. The method's applicability across multiple benchmarks suggests generalizability beyond specific architectures. As MLLMs become increasingly embedded in enterprise applications requiring factual accuracy—from medical imaging to document analysis—robust hallucination mitigation becomes commercially critical. The work establishes a pathway toward more reliable multimodal systems, potentially accelerating enterprise adoption.

• →DeP uses dynamic textual perturbations during decoding to expose and counteract language priors without model retraining
• →The method preserves model fluency while improving factual grounding across multiple benchmarks
• →Attention variance analysis identifies stable evidence regions and suppresses noise in feature representations
• →Training-free approach reduces deployment friction compared to fine-tuning-based solutions
• →Framework applicable to addressing MLLM reliability in production environments requiring high factual accuracy