COHERENCE: Benchmarking Fine-Grained Image-Text Alignment in Interleaved Multimodal Contexts

Researchers introduced COHERENCE, a new benchmark for evaluating Multimodal Large Language Models (MLLMs) on their ability to understand fine-grained image-text alignment in interleaved contexts—such as documents with mixed text and images. The benchmark contains 6,161 high-quality questions across four domains and includes error analysis to identify specific capability gaps in current models.

COHERENCE addresses a critical gap in MLLM evaluation methodology. While existing benchmarks excel at measuring single-image comprehension and isolated multi-image tasks, they fail to capture real-world complexity where text and images appear intermixed—the norm in documents, web pages, and technical materials. This benchmark represents progress toward more realistic assessment frameworks.

The research reflects broader industry recognition that benchmark sophistication must evolve alongside model capabilities. As MLLMs become commodity tools integrated into production systems, evaluation frameworks need to measure practical competencies rather than isolated tasks. Document understanding remains a high-value application, yet most models show weakness when images and text require tight semantic coupling.

The six-type error analysis distinguishes COHERENCE from previous work by enabling diagnostic insights. Rather than reporting aggregate scores, developers gain granular attribution of failures—whether models struggle with visual recognition, textual comprehension, alignment between modalities, or reasoning over interleaved signals. This diagnostic capability accelerates targeted model improvements.

For developers building document processing systems, this benchmark provides a standardized evaluation methodology to assess MLLM suitability. For researchers, COHERENCE establishes quantitative baselines for measuring progress on interleaved multimodal understanding. The 6,161-question dataset spanning multiple domains creates sufficient diversity to reveal model weaknesses across different content types and visual-textual relationships.

• →COHERENCE benchmark evaluates MLLMs on fine-grained image-text alignment in realistic interleaved multimodal contexts.
• →The benchmark contains 6,161 high-quality questions across four domains with six-type error analysis for diagnostic insights.
• →Existing MLLM benchmarks underrepresent document-style interleaved content, creating evaluation blind spots for production use cases.
• →Six-category error analysis enables specific attribution of MLLM failures to missing capabilities rather than aggregate scoring.
• →This work advances evaluation methodology to match the increasing sophistication of deployed multimodal AI systems.