AIBearisharXiv – CS AI · May 277/10
🧠Researchers introduce VisualNeedle, a benchmark that exposes limitations in multimodal large language models' ability to perform genuine fine-grained visual search in information-dense scenes. Despite frontier MLLMs reporting over 90% accuracy on existing benchmarks, VisualNeedle reveals that these models struggle significantly when critical evidence is spatially constrained to minute regions, with the best model achieving only 56% accuracy versus 63% human performance.
AIBullisharXiv – CS AI · Apr 147/10
🧠Researchers introduce SpatialScore, a comprehensive benchmark with 5K samples across 30 tasks to evaluate multimodal language models' spatial reasoning capabilities. The work includes SpatialCorpus, a 331K-sample training dataset, and SpatialAgent, a multi-agent system with 12 specialized tools, demonstrating significant improvements in spatial intelligence without additional model training.
AINeutralarXiv – CS AI · Jun 196/10
🧠Researchers introduced ROSE, a benchmark that evaluates how well multimodal language models can convert visual information into context-specific actions. Testing nine MLLMs revealed significant performance drops of up to 44.5 percentage points when shifting from counting tasks to region-conditioned actions, despite near-perfect human performance, indicating a fundamental gap in how these models translate perception into actionable outputs.
AINeutralarXiv – CS AI · Jun 16/10
🧠Researchers introduce ERGeoBench, a comprehensive benchmark for evaluating multimodal large language models (MLLMs) on embodied geo-localization tasks using 2,207 street-view panoramas across three progressive difficulty settings. The evaluation reveals that current leading models can understand high-level geographic semantics but struggle with fine-grained perception, metric localization, and spatial consistency, highlighting that accurate geo-localization requires integrated perception and reasoning rather than isolated visual recognition.
AINeutralarXiv – CS AI · May 296/10
🧠Researchers introduce GUITestScape, a new benchmark for evaluating AI agents' ability to autonomously test Android applications, along with GUIJudge, an evaluator that assesses both interaction and display defects beyond predefined annotations. The work addresses critical gaps in current GUI testing evaluation by enabling process-aware assessment of agent capabilities rather than just final outcomes.
AINeutralarXiv – CS AI · May 286/10
🧠Researchers introduce MMTABREAL, a new benchmark dataset of 500 real-world multimodal tables with 4,021 question-answer pairs designed to rigorously evaluate how well AI language models understand tables containing charts, maps, icons, and color encodings. Testing reveals significant performance gaps in state-of-the-art models, particularly in visual grounding and multi-step reasoning, indicating that current architectures lack tight fusion between vision and tabular structure.
AINeutralarXiv – CS AI · May 276/10
🧠Researchers introduced OCR-Reasoning, a new benchmark with 1,069 annotated examples to evaluate how well multimodal AI models handle text-rich image reasoning tasks. The evaluation revealed that even the most advanced models fail to exceed 50% accuracy, indicating significant gaps in this critical capability area.
AINeutralarXiv – CS AI · May 126/10
🧠Researchers introduce BenchCAD, a comprehensive benchmark containing 17,900 execution-verified CAD programs across 106 industrial part families, designed to evaluate multimodal AI models on their ability to generate parametric CAD code from visual or textual inputs. Testing 10+ frontier models reveals that current systems can recover basic geometry but struggle with faithful parametric abstraction, fine 3D structure, and complex CAD operations, highlighting significant gaps between general-purpose AI capabilities and industrial CAD automation readiness.
AINeutralarXiv – CS AI · May 96/10
🧠Researchers propose Vision-Language Logical Consistency Metric (VL-LCM), a novel evaluation framework for multimodal large language models that assesses logical coherence without requiring ground-truth annotations. Testing 11 MLLMs across benchmarks including MMMU and NaturalBench reveals that while accuracy has improved significantly, logical consistency substantially lags, suggesting current models make confident but logically inconsistent predictions.
AINeutralarXiv – CS AI · May 16/10
🧠Researchers introduced COHERENCE, a new benchmark for evaluating Multimodal Large Language Models (MLLMs) on their ability to understand fine-grained image-text alignment in interleaved contexts—such as documents with mixed text and images. The benchmark contains 6,161 high-quality questions across four domains and includes error analysis to identify specific capability gaps in current models.
AINeutralarXiv – CS AI · Apr 136/10
🧠Researchers introduce AV-SpeakerBench, a new 3,212-question benchmark designed to evaluate how well multimodal large language models understand audiovisual speech by correlating speakers with their dialogue and timing. Testing reveals Gemini 2.5 Pro significantly outperforms open-source competitors, with the gap primarily attributable to inferior audiovisual fusion capabilities rather than visual perception limitations.
🧠 Gemini
