Reasoning Matters: Mitigate Hallucination in Multimodal Large Reasoning Models via Reasoning-Conditioned Preference Optimization
Researchers propose Reasoning-Conditioned Direct Preference Optimization (RC-DPO), a training method that reduces hallucinations in multimodal large reasoning models by treating chain-of-thought reasoning as a condition for answer generation rather than a monolithic output. The approach uses Monte Carlo Tree Search to generate better training data and demonstrates improved reliability across multiple benchmarks.
This research addresses a critical vulnerability in multimodal large reasoning models—their propensity to generate plausible-sounding but incorrect outputs, or hallucinations. The work reveals that existing direct preference optimization (DPO) methods treat reasoning chains and final answers as inseparable units, inadvertently optimizing primarily for answer-level preferences while underutilizing chain-of-thought supervision. This finding has significant implications for AI safety and reliability.
The RC-DPO framework separates these components conceptually, modeling the chain-of-thought as a conditioning variable that influences answer generation. By contrasting preferences for identical answers derived from different reasoning paths, the method encourages models to learn answer-supportive reasoning patterns. The researchers enhance this with Monte Carlo Tree Search to discover visually grounded and logically consistent reasoning chains as positive examples, while using attention-guided pruning to construct challenging negative examples.
For developers building vision-language AI systems, this represents a meaningful advance in mitigating hallucinations—a problem that undermines production deployment of multimodal models. The method's effectiveness across multiple models and benchmarks suggests broad applicability rather than narrow optimization. This becomes particularly important as reasoning-capable multimodal models see increased adoption in high-stakes applications.
The work establishes a foundation for more rigorous reasoning alignment in AI systems. Future directions likely involve scaling these techniques to larger models and exploring additional conditioning strategies. As multimodal AI systems become more prevalent in enterprise and consumer applications, improvements in hallucination mitigation directly impact user trust and system reliability.
- →RC-DPO treats chain-of-thought as a condition for answer generation, improving alignment between reasoning and outputs
- →Existing DPO methods primarily optimize answer preferences while underutilizing reasoning supervision
- →Monte Carlo Tree Search discovers visually grounded reasoning chains to enhance training data quality
- →The method demonstrates broad effectiveness across multiple models, indicating practical deployment viability
- →Hallucination reduction in reasoning models improves reliability for high-stakes applications