IMUG-Bench: Benchmarking Unified Multimodal Models on Interleaved Understanding and Generation

Researchers introduce IMUG-Bench, a comprehensive benchmark designed to evaluate unified multimodal models (UMMs) on their ability to handle multi-turn interleaved image-text dialogues. The benchmark reveals that current models struggle with exposure bias in generation tasks and that test-time scaling strategies like Chain-of-Thought can improve performance.

The emergence of unified multimodal models represents a significant evolution in AI capabilities, as these systems attempt to handle both understanding and generation tasks within a single framework. IMUG-Bench addresses a critical gap in the evaluation landscape by focusing on multi-turn interactions, which more closely reflect real-world usage patterns than existing single-turn benchmarks. This matters because production AI systems must maintain coherence and accuracy across extended conversations involving both image and text inputs.

The research builds on years of progress in multimodal AI, where models like GPT-4V and Gemini have demonstrated impressive capabilities. However, the field has lacked standardized evaluation methods for the specific challenge of maintaining performance across dynamic, context-dependent exchanges. Exposure bias—where models trained on ground-truth inputs encounter their own outputs during inference—becomes increasingly problematic in multi-turn settings, yet few benchmarks explicitly measure this phenomenon.

The benchmark's findings have direct implications for developers and organizations deploying multimodal systems. The discovery that mainstream models exhibit pronounced exposure bias suggests current architectures may degrade unpredictably in extended conversations, a critical concern for customer-facing applications. The 3,113 samples spanning 12,034 interaction turns provide a robust evaluation foundation that developers can use to stress-test their systems before production deployment.

Looking ahead, the demonstrated effectiveness of test-time scaling strategies offers practical improvements without requiring model retraining. Organizations implementing multimodal dialogue systems should prioritize evaluation against multi-turn benchmarks and consider deploying inference-time techniques to mitigate identified failure modes. Future research will likely focus on architectural innovations that address exposure bias at the model level rather than relying solely on post-hoc corrections.

• →IMUG-Bench introduces the first comprehensive benchmark specifically designed for multi-turn interleaved image-text dialogue evaluation.
• →Large-scale experiments reveal that unified multimodal models exhibit significant exposure bias when generating responses across multiple turns.
• →Test-time scaling strategies including Chain-of-Thought and Best-of-N Sampling effectively mitigate exposure bias and improve generation accuracy.
• →The benchmark covers 3,113 samples across three dialogue classes (Static Spatial, Temporal Causal, Hybrid) reflecting real-world interaction complexity.
• →Current open-source and closed-source UMMs show clear capability boundaries that developers should understand before production deployment.