Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs

Researchers introduced 'Mind's Eye,' a benchmark that tests multimodal large language models (MLLMs) on visual reasoning tasks inspired by human intelligence tests. The evaluation reveals a significant gap between human performance (80% accuracy) and leading MLLMs (below 50%), exposing limitations in visuospatial reasoning, visual attention, and conceptual abstraction.

The introduction of 'Mind's Eye' addresses a critical gap in how we evaluate modern AI systems. While MLLMs have demonstrated impressive capabilities on standard vision-language benchmarks, this research reveals they struggle with core cognitive tasks that humans find relatively straightforward. The benchmark's A-R-T taxonomy—covering Abstraction, Relation, and Transformation—directly targets fluid intelligence processes essential for genuine visual understanding rather than pattern matching in training data.

This work builds on growing recognition that benchmark saturation masks real limitations in AI reasoning. As MLLMs achieve near-human performance on existing benchmarks, researchers increasingly turn to more cognitively grounded evaluations. The 30-percentage-point gap between human and MLLM performance suggests current architectures lack mechanisms for internal perceptual manipulation and abstract visual concept formation—capabilities that don't naturally emerge from scaling alone.

For the AI research community, these findings carry important implications. They suggest that simply scaling model size or training data won't bridge the visuospatial reasoning gap; architectural innovations targeting fluid intelligence may be necessary. The error analysis identifying failures in visual attention allocation and abstraction provides actionable guidance for researchers developing next-generation models. Developers building applications requiring spatial reasoning or visual problem-solving cannot rely on current MLLMs without significant limitations.

• →Top MLLMs achieve below 50% accuracy on visuospatial reasoning tasks where humans score 80%.
• →The benchmark reveals three critical failure modes: poor visual attention, weak internal perceptual manipulation, and limited abstraction capabilities.
• →Current evaluation frameworks may mask fundamental limitations in AI cognitive abilities.
• →Addressing these gaps likely requires architectural innovations beyond scaling existing approaches.
• →Cognitively grounded benchmarks provide more meaningful measures of AI progress than saturated standard benchmarks.