AMVICC: A Novel Benchmark for Cross-Modal Failure Mode Profiling for VLMs and IGMs

Researchers introduce AMVICC, a novel benchmark for evaluating failure modes in vision-language models (VLMs) and image generation models (IGMs). Testing 11 multimodal LLMs and 3 IGMs across 9 visual reasoning categories, the study reveals that both model types struggle with basic visual concepts like object orientation, quantity, and spatial relationships, with some failures shared across modalities and others model-specific.

The AMVICC benchmark represents a systematic attempt to understand fundamental weaknesses in vision-language AI systems at a time when these models are increasingly deployed in real-world applications. By creating a cross-modal evaluation framework that tests both image-to-text and text-to-image capabilities, researchers identified critical gaps in elementary visual reasoning that persist despite rapid advances in model scale and training data.

This research builds on the growing recognition that larger models do not automatically solve reasoning problems. The MMVP benchmark adaptation methodology allows researchers to probe both explicit and implicit understanding of visual concepts, revealing that image generation models particularly struggle with fine-grained attribute control. This distinction between shared and model-specific failures suggests that different architectures have fundamentally different knowledge representations.

For the AI development community, these findings highlight that unified vision-language approaches require careful attention to visual grounding beyond token prediction. Current MLLMs and IGMs operate with distinct failure patterns despite shared training objectives, indicating that cross-modal alignment remains an unsolved problem. Developers building applications requiring precise visual understanding—such as robotics, medical imaging, or quality control systems—cannot rely on current models for tasks involving spatial reasoning or quantitative visual analysis.

The framework established by AMVICC provides a foundation for future research into whether image generation and interpretation failures stem from shared architectural limitations or training data gaps. This knowledge will likely influence how researchers design next-generation unified vision-language models that can maintain consistency across modalities.

• →Vision-language models consistently fail at basic visual reasoning tasks including object orientation, quantity assessment, and spatial relationship understanding.
• →Image generation models show particularly poor fine-grained control over visual attributes in response to explicit prompts.
• →Failure modes are partially shared between models and modalities but also exhibit model-specific and modality-specific patterns.
• →AMVICC benchmark enables systematic cross-modal evaluation to identify whether failures stem from shared architectural limitations.
• →Current unified vision-language approaches require significant improvements for reliable deployment in precision-dependent applications.