Omni-Perception Policy Optimization for Multimodal Emotion Reasoning

Researchers introduce OPPO, a reinforcement learning framework designed to improve how multimodal AI systems (Omni-MLLMs) understand emotion by better integrating visual, acoustic, and textual information. The method addresses critical failures where systems hallucinate cross-modal information and fail to fully utilize available data, achieving state-of-the-art results on emotion recognition benchmarks.

This research addresses a fundamental limitation in multimodal large language models tasked with emotion recognition and reasoning. Current systems struggle with two critical problems: they inadequately leverage the complementary information available across different modalities (visual expressions, voice tone, text content), and they suffer from hallucination where they fabricate modality-specific details from information not present in certain input channels. These failures undermine the reliability of AI systems used in mental health applications, customer service analytics, and content recommendation systems.

The OPPO framework tackles these issues through a dual-mechanism approach. The Omni-Perception Reward mechanism decomposes reasoning processes into granular visual, acoustic, and emotional components, incentivizing the model to explicitly recover these distinct cues rather than relying on shortcuts. Simultaneously, the Omni-Perception Loss function prevents hallucination by comparing model behavior under full multimodal inputs against scenarios where individual modalities are masked, penalizing the model when it generates modality-specific claims unsupported by actual input data.

The introduction of MEP-Bench provides quantifiable metrics for assessing both utilization and faithfulness, enabling systematic evaluation beyond traditional emotion recognition accuracy. The framework's strong performance improvements across MER-UniBench and MME-Emotion benchmarks demonstrate that explicit multimodal perception optimization yields measurable gains.

For the broader AI industry, this work validates that foundational robustness challenges in multimodal reasoning require specialized training objectives rather than scale alone. As emotion AI applications expand into high-stakes domains, these reliability improvements become increasingly critical for deployment trust and regulatory compliance.

• →OPPO uses reinforcement learning to explicitly optimize multimodal perception, addressing underutilization and hallucination in emotion-reasoning AI systems
• →Dual-mechanism approach combines fine-grained reward decomposition with KL-penalized loss to suppress cross-modal hallucination while improving utilization
• →MEP-Bench diagnostic benchmark provides quantifiable metrics for evaluating both utilization and faithfulness in multimodal emotion reasoning
• →Framework achieves state-of-the-art performance on MER-UniBench and MME-Emotion while substantially improving reliability metrics
• →Research demonstrates that specialized training objectives addressing specific multimodal reasoning failures are more effective than general scaling approaches