AIBearisharXiv – CS AI · 4d ago7/10
🧠GlobalDentBench introduces the first multinational dental benchmark with 8,978 expert-validated questions across 14 specialties, revealing that current LLMs face severe limitations in clinical reasoning with a 31.01% unsafe recommendation rate. The study demonstrates performance degrades sharply as reasoning complexity increases, with accuracy dropping from 81.34% on multiple-choice to just 22.34% on case-based questions, highlighting critical safety gaps before LLMs can be deployed in healthcare.
AIBearisharXiv – CS AI · May 127/10
🧠Researchers expose critical flaws in Computer Use Agent (CUA) benchmarking, demonstrating that simple replay scripts outperform advanced AI models on current static benchmarks. The study introduces PRISM design principles and DigiWorld, a rigorous evaluation framework with 3.2 million verified configurations, establishing new standards for meaningful CUA assessment.
AIBullisharXiv – CS AI · May 117/10
🧠Researchers introduce APEX, a novel image quality assessment metric that addresses fundamental limitations in existing evaluation methods like FID by using Sliced Wasserstein Distance and modern foundation models (CLIP, DINOv2) as embedding-agnostic feature extractors. The framework eliminates parametric assumptions while maintaining scalability to high-dimensional spaces, demonstrating superior robustness and stability across datasets.
AINeutralarXiv – CS AI · May 117/10
🧠Researchers have developed a psychometric framework to evaluate generative AI models' cognitive abilities across generations, revealing profound imbalances in their intelligence architecture. While leading multimodal models excel at verbal comprehension and working memory (>98th percentile), they severely lag in perceptual reasoning (<1st percentile), indicating that scaling alone cannot achieve human-like general intelligence.
AINeutralarXiv – CS AI · May 47/10
🧠TokenArena introduces a continuous benchmark framework that evaluates AI inference endpoints across energy efficiency, latency, cost, and output quality rather than just model-level comparisons. Testing 78 endpoints across 12 model families reveals dramatic performance variance—the same model differs by up to 12.5 accuracy points and 6.2x in energy efficiency depending on deployment configuration, with workload type fundamentally reordering cost-effectiveness rankings.
AIBullisharXiv – CS AI · May 47/10
🧠Researchers developed Legal Assist AI, a framework using an 8-billion-parameter Llama 3.1 model enhanced with Retrieval-Augmented Generation to provide legal assistance tailored to Indian law. The system achieved 60.08% on the All-India Bar Examination benchmark, outperforming OpenAI's 175-billion-parameter GPT-3.5 Turbo while being 22 times more parameter-efficient.
🧠 Llama
AINeutralarXiv – CS AI · Apr 207/10
🧠Researchers introduced MEDLEY-BENCH, a new AI benchmark that evaluates metacognition—an AI model's ability to monitor and revise its own reasoning. The study found that while larger models evaluate their reasoning better, they don't actually control their outputs more effectively, and smaller models often match larger ones in metacognitive tasks, suggesting scale alone doesn't determine reasoning quality.
AINeutralarXiv – CS AI · Apr 207/10
🧠Researchers introduced PRL-Bench, a comprehensive benchmark measuring large language models' ability to conduct autonomous physics research across five subfields. Testing frontier AI models revealed performance below 50%, exposing a significant capability gap between current LLMs and the demands of real-world scientific discovery.
AINeutralarXiv – CS AI · Apr 147/10
🧠Researchers introduce PaperScope, a comprehensive benchmark for evaluating multi-modal AI systems on complex scientific research tasks across multiple documents. The benchmark reveals that even advanced systems like OpenAI Deep Research and Tongyi Deep Research struggle with long-context retrieval and cross-document reasoning, exposing significant gaps in current AI capabilities for scientific workflows.
🏢 OpenAI
AIBearisharXiv – CS AI · Apr 107/10
🧠Researchers reveal that Large Language Models exhibit self-preference bias when evaluating other LLMs, systematically favoring outputs from themselves or related models even when using objective rubric-based criteria. The bias can reach 50% on objective benchmarks and 10-point score differences on subjective medical benchmarks, potentially distorting model rankings and hindering AI development.
AINeutralarXiv – CS AI · Mar 177/10
🧠Researchers introduce the AI Transformation Gap Index (AITG), the first empirical framework to measure firms' AI readiness relative to competitors and translate it into quantifiable financial outcomes. The framework analyzes 22 industries and shows that larger AI transformation gaps don't always create the highest value due to implementation challenges and timing issues.
AINeutralarXiv – CS AI · Mar 57/10
🧠Researchers introduce SpatialBench, a comprehensive benchmark for evaluating spatial cognition in multimodal large language models (MLLMs). The framework reveals that while MLLMs excel at perceptual grounding, they struggle with symbolic reasoning, causal inference, and planning compared to humans who demonstrate more goal-directed spatial abstraction.
AINeutralarXiv – CS AI · Mar 46/103
🧠Researchers introduce CFE-Bench, a new multimodal benchmark for evaluating AI reasoning across 20+ STEM domains using authentic university exam problems. The best performing model, Gemini-3.1-pro-preview, achieved only 59.69% accuracy, highlighting significant gaps in AI reasoning capabilities, particularly in maintaining correct intermediate states through multi-step solutions.
AINeutralarXiv – CS AI · Mar 37/103
🧠Researchers introduce InnoGym, the first benchmark designed to evaluate AI agents' innovation potential rather than just correctness. The framework measures both performance gains and methodological novelty across 18 real-world engineering and scientific tasks, revealing that while AI agents can generate novel approaches, they lack robustness for significant performance improvements.
AINeutralarXiv – CS AI · 2d ago6/10
🧠Researchers introduced AtomWorld, a benchmark for evaluating how well large language models can perform spatial reasoning tasks in materials science, specifically atomic structure manipulation. The study reveals that current LLMs like Claude Opus 4.6 struggle with complex spatial operations, achieving success rates below 12% for rotation tasks, suggesting they function better as collaborative tools than autonomous scientific agents.
🧠 Claude🧠 Opus
AINeutralarXiv – CS AI · 2d ago6/10
🧠The BEAMS Initiative establishes benchmarks to evaluate AI tools for modeling and simulation, ensuring they complement human expertise rather than replace it. Testing reveals that current AI-enabled modeling tools excel at discussion and qualitative tasks but struggle with causal reasoning and quantitative error correction, with performance varying significantly across different LLM implementations.
AINeutralarXiv – CS AI · 3d ago6/10
🧠Researchers introduced SpatialBench-Long, a comprehensive benchmark testing AI agents' ability to conduct end-to-end scientific reasoning on complex spatial biology data without prescribed methods. The benchmark spans 24 evaluations across multiple cancer and aging systems using diverse measurement technologies, with current leading models achieving only 11.1% success rate, revealing significant limitations in AI's capacity for autonomous biological discovery.
🏢 OpenAI🧠 GPT-5🧠 Gemini
AINeutralarXiv – CS AI · 3d ago6/10
🧠Researchers argue that current AI evaluation benchmarks fail to reflect real-world performance in low-resource environments, where factors like noisy inputs, poor connectivity, and low-end hardware significantly impact usability. The paper proposes a new evaluation framework that assesses deployed systems holistically rather than isolated models, with standardized reporting cards designed for policymakers and implementers.
AINeutralarXiv – CS AI · 3d ago6/10
🧠Researchers evaluated whether large language models can realistically simulate human behavior in online discourse by comparing LLM-generated reactions to Spanish news articles against real audience responses across hate speech, sentiment, and semantic alignment metrics. The study found that off-the-shelf models significantly underreproduce hate speech and introduce model-specific biases, while fine-tuning improves fidelity unevenly depending on the model.
AIBearisharXiv – CS AI · 3d ago6/10
🧠Researchers introduce DynaSchedBench, a calibrated framework for testing AI agents on dynamic job scheduling problems, revealing that large language models underperform expectations. The study uncovers an 'Observability Paradox' where providing agents with complete information actually degrades performance, and shows LLM-based schedulers fail to consistently outperform traditional heuristic baselines despite significant computational overhead.
AINeutralarXiv – CS AI · 4d ago6/10
🧠Researchers introduce Drive-P2D, a comprehensive benchmark for evaluating vision-language models in autonomous driving that tests perception and decision-making across progressive complexity levels. The benchmark addresses gaps in existing evaluation methods by separating reasoning analysis from objective answer scoring and identifying specific failure modes that could improve VLM safety for real-world deployment.
AINeutralarXiv – CS AI · May 126/10
🧠The CODS 2025 AssetOpsBench competition retrospective reveals critical gaps between public and private evaluation metrics in multi-agent orchestration systems. Hidden test sets dramatically altered performance rankings, particularly in execution tasks where correlations turned negative, while successful teams prioritized guardrails over novel architectures.
AINeutralarXiv – CS AI · May 116/10
🧠LithoBench introduces a comprehensive benchmark dataset for evaluating large multimodal models on remote-sensing lithology interpretation, containing 10,000 expert-annotated instances across cognitive levels from identification to reasoning. The research reveals significant gaps in current vision-language models' ability to handle knowledge-intensive geological tasks, highlighting the challenges of applying general-purpose AI to specialized domain expertise.
AIBullisharXiv – CS AI · May 116/10
🧠Researchers compared frontier Large Reasoning Models (LRMs) with traditional AI systems using human gameplay data paired with fMRI brain recordings. LRMs demonstrated superior alignment with human learning behavior and predicted brain activity an order of magnitude better than reinforcement learning alternatives, suggesting they more closely mirror human cognition during complex decision-making.
AINeutralarXiv – CS AI · May 116/10
🧠A comprehensive eight-week study evaluated 68 HTML generations from four major LLM families (GPT, Gemini, Grok, Claude) in standardized web generation tasks, finding Claude delivered the most consistent performance while questioning assumptions about reasoning time and social media predictability. The research reveals significant evaluation bias in LLM-as-judge systems and that code verbosity correlates more with model architecture than prompt specificity.
🧠 Claude🧠 Gemini🧠 Grok
