MVCL-DAF++: Enhancing Multimodal Intent Recognition via Prototype-Aware Contrastive Alignment and Coarse-to-Fine Dynamic Attention Fusion

Researchers introduce MVCL-DAF++, an advanced multimodal intent recognition system that combines prototype-aware contrastive alignment with coarse-to-fine dynamic attention fusion to improve semantic understanding and robustness. The model achieves state-of-the-art performance on benchmark datasets, with notable improvements in rare-class recognition accuracy.

MVCL-DAF++ addresses fundamental challenges in multimodal machine learning where systems must integrate information across text, audio, and visual modalities while maintaining semantic coherence. Traditional approaches struggle with weak semantic grounding and performance degradation when encountering noisy data or underrepresented classes. This research tackles these limitations through two complementary mechanisms: prototype-aware contrastive alignment grounds individual instances to learned class-level representations, strengthening semantic consistency across modalities, while coarse-to-fine attention fusion hierarchically combines global modality summaries with token-level granular features to capture both broad context and detailed cross-modal interactions.

The reported improvements on MIntRec benchmarks—with rare-class recognition gains of 1.05% and 4.18% weighted F1 scores—demonstrate that prototype-guided learning effectively enhances model robustness in practical, imbalanced scenarios. These benchmarks represent real-world conditions where certain user intents appear infrequently in training data, making rare-class performance a critical metric for production systems.

For the AI research community and practitioners developing multimodal systems, this work provides actionable architectural insights applicable to dialogue systems, voice assistants, and autonomous agents that must understand user intent from multiple information streams. The public code release enables rapid adoption and builds toward more reliable multimodal AI systems. The demonstrated effectiveness of prototype-based learning suggests broader applicability beyond intent recognition to other multimodal understanding tasks requiring semantic robustness.

• →MVCL-DAF++ combines prototype-aware contrastive alignment with coarse-to-fine attention fusion for enhanced multimodal understanding.
• →Model achieves state-of-the-art results on MIntRec and MIntRec2.0 benchmarks with significant rare-class recognition improvements.
• →Prototype-guided learning strengthens semantic consistency by aligning instances to class-level representations across modalities.
• →Hierarchical attention fusion integrates global and token-level features for more effective cross-modal interaction.
• →Source code availability enables broader adoption and application to production multimodal AI systems.