AINeutralarXiv – CS AI · May 116/10
🧠LithoBench introduces a comprehensive benchmark dataset for evaluating large multimodal models on remote-sensing lithology interpretation, containing 10,000 expert-annotated instances across cognitive levels from identification to reasoning. The research reveals significant gaps in current vision-language models' ability to handle knowledge-intensive geological tasks, highlighting the challenges of applying general-purpose AI to specialized domain expertise.
AINeutralarXiv – CS AI · May 46/10
🧠Researchers introduce LEGIT, a 24K-instance legal reasoning dataset with hierarchical argument trees that serve as evaluation rubrics for LLM-generated legal reasoning. The study reveals that LLM legal reasoning performance depends critically on both issue coverage and correctness, with RAG and reinforcement learning offering complementary improvements.
AINeutralarXiv – CS AI · May 16/10
🧠Researchers evaluated eight LLM agents across three interaction paradigms—domain-specific agents, computer-use agents, and general-purpose coding agents—on scientific visualization tasks. The study reveals fundamental tradeoffs: general-purpose agents excel at task completion but consume more computational resources, while domain-specific agents offer efficiency and stability at the cost of flexibility, with persistent memory improving performance across modalities.
AIBearisharXiv – CS AI · May 16/10
🧠A comprehensive study comparing 12 large language models against 4 classical classifiers for automating evidence screening in software engineering systematic literature reviews reveals that LLMs exhibit significant performance variability and lack consistent superiority over traditional methods. The research emphasizes that abstract availability is critical for LLM performance, while title and keywords provide minimal additional value, suggesting LLM adoption should be driven by operational constraints rather than performance guarantees.
🏢 OpenAI🏢 Anthropic🧠 Gemini
AINeutralarXiv – CS AI · Apr 156/10
🧠The first LLM Testing competition at ICSE 2026's DeepTest workshop evaluated four tools designed to benchmark an LLM-based automotive assistant, focusing on their ability to identify failure cases where the system fails to surface critical safety warnings from car manuals. The competition assessed both the effectiveness of test discovery and the diversity of identified failures, establishing a benchmark for evaluating AI testing methodologies in safety-critical applications.
AINeutralarXiv – CS AI · Apr 146/10
🧠Researchers have released LABBench2, an upgraded benchmark with nearly 1,900 tasks designed to measure AI systems' real-world capabilities in biology research beyond theoretical knowledge. The new benchmark shows current frontier models achieve 26-46% lower accuracy than on the original LAB-Bench, indicating significant progress in AI scientific abilities while highlighting substantial room for improvement.
$OP🏢 Hugging Face
AINeutralarXiv – CS AI · Apr 146/10
🧠Researchers introduced COMPOSITE-STEM, a new benchmark containing 70 expert-written scientific tasks across physics, biology, chemistry, and mathematics to evaluate AI agents. The top-performing model achieved only 21% accuracy, indicating the benchmark effectively measures capabilities beyond current AI reach and addresses the saturation of existing evaluation frameworks.
AINeutralarXiv – CS AI · Apr 146/10
🧠RPA-Check introduces an automated four-stage framework for evaluating Large Language Model-based Role-Playing Agents in complex scenarios, addressing the gap in standard NLP metrics for assessing role adherence and narrative consistency. Testing across legal scenarios reveals that smaller, instruction-tuned models (8-9B parameters) outperform larger models in procedural consistency, suggesting optimal performance doesn't correlate with model scale.
AINeutralarXiv – CS AI · Apr 136/10
🧠Researchers introduce Spatial-Gym, a benchmarking environment that evaluates AI models on spatial reasoning tasks through step-by-step pathfinding in 2D grids rather than one-shot generation. Testing eight models reveals a significant performance gap, with the best model achieving only 16% solve rate versus 98% for humans, exposing critical limitations in how AI systems scale reasoning effort and process spatial information.
AINeutralarXiv – CS AI · Apr 136/10
🧠Researchers introduce ReplicatorBench, a comprehensive benchmark for evaluating AI agents' ability to replicate scientific research claims in social and behavioral sciences. The study reveals that current LLM agents excel at designing and executing experiments but struggle significantly with data retrieval, highlighting critical gaps in autonomous research validation capabilities.
AINeutralarXiv – CS AI · Apr 76/10
🧠Researchers introduce ClawArena, a new benchmark for evaluating AI agents' ability to maintain accurate beliefs in evolving information environments with conflicting sources. The benchmark tests 64 scenarios across 8 professional domains, revealing significant performance gaps between different AI models and frameworks in handling dynamic belief revision and multi-source reasoning.
AINeutralarXiv – CS AI · Mar 266/10
🧠Researchers developed a method to evaluate AI agents more efficiently by testing them on only 30-44% of benchmark tasks, focusing on mid-difficulty problems. The approach maintains reliable rankings while significantly reducing computational costs compared to full benchmark evaluation.
AINeutralarXiv – CS AI · Mar 176/10
🧠NetArena introduces a dynamic benchmarking framework for evaluating AI agents in network automation tasks, addressing limitations of static benchmarks through runtime query generation and network emulator integration. The framework reveals that AI agents achieve only 13-38% performance on realistic network queries, significantly improving statistical reliability by reducing confidence-interval overlap from 85% to 0%.
AINeutralarXiv – CS AI · Mar 166/10
🧠Research reveals that large language models used as judges for scoring responses show misleading performance when evaluated by global correlation metrics versus actual best-of-n selection tasks. A study using 5,000 prompts found that judges with moderate global correlation (r=0.47) only captured 21% of potential improvement, primarily due to poor within-prompt ranking despite decent overall agreement.
AINeutralarXiv – CS AI · Mar 166/10
🧠SkillsBench introduces a new benchmark to evaluate Agent Skills - structured packages of procedural knowledge that enhance LLM agents. Testing across 86 tasks and 11 domains shows curated Skills improve performance by 16.2 percentage points on average, while self-generated Skills provide no benefit.
AINeutralarXiv – CS AI · Mar 96/10
🧠Researchers introduced VisioMath, a new benchmark with 1,800 K-12 math problems designed to test Large Multimodal Models' ability to distinguish between visually similar diagrams. The study reveals that current state-of-the-art models struggle with fine-grained visual reasoning, often relying on shallow positional heuristics rather than proper image-text alignment.
AINeutralarXiv – CS AI · Mar 96/10
🧠Researchers introduce KramaBench, a comprehensive benchmark testing AI systems' ability to execute end-to-end data processing pipelines on real-world data lakes. The study reveals significant limitations in current AI systems, with the best performing system achieving only 55% accuracy in full data-lake scenarios and leading LLMs implementing just 20% of individual data tasks correctly.
AIBullisharXiv – CS AI · Mar 36/108
🧠Researchers introduce M-JudgeBench, a comprehensive benchmark for evaluating Multimodal Large Language Models (MLLMs) used as judges, and propose Judge-MCTS framework to improve judge model training. The work addresses systematic weaknesses in existing MLLM judge systems through capability-oriented evaluation and enhanced data generation methods.
AINeutralarXiv – CS AI · Mar 36/107
🧠Researchers introduced Pencil Puzzle Bench, a new framework for evaluating large language model reasoning capabilities using constraint-satisfaction problems. The benchmark tested 51 models across 300 puzzles, revealing significant performance improvements through increased reasoning effort and iterative verification processes.
AIBullisharXiv – CS AI · Mar 37/107
🧠Researchers introduce CARE, a new framework for improving LLM evaluation by addressing correlated errors in AI judge ensembles. The method separates true quality signals from confounding factors like verbosity and style preferences, achieving up to 26.8% error reduction across 12 benchmarks.
AINeutralarXiv – CS AI · Mar 37/106
🧠Researchers introduce MOSAIC, the first comprehensive benchmark to evaluate moral, social, and individual characteristics of Large Language Models beyond traditional Moral Foundation Theory. The benchmark includes over 600 curated questions and scenarios from nine validated questionnaires and four platform-based games, providing empirical evidence that current evaluation methods are insufficient for assessing AI ethics comprehensively.
AINeutralarXiv – CS AI · Mar 36/103
🧠Researchers introduce FaithCoT-Bench, the first comprehensive benchmark for detecting unfaithful Chain-of-Thought reasoning in large language models. The benchmark includes over 1,000 expert-annotated trajectories across four domains and evaluates eleven detection methods, revealing significant challenges in identifying unreliable AI reasoning processes.
AINeutralarXiv – CS AI · Mar 27/1017
🧠Researchers conducted a benchmark study on IoT botnet intrusion detection systems, finding that models trained on one network domain suffer significant performance degradation when applied to different environments. The study evaluated three feature sets across four IoT datasets and provided guidelines for improving cross-domain robustness through better feature engineering and algorithm selection.
AIBullishGoogle DeepMind Blog · Oct 236/106
🧠Game Arena is a new open-source platform designed for rigorous AI model evaluation, enabling direct head-to-head comparisons of frontier AI systems in competitive environments with clear victory conditions. This represents a shift toward more standardized and comparative methods for measuring AI intelligence and capabilities.
AIBullishHugging Face Blog · Jun 66/105
🧠Artificial Analysis has launched a new Text to Image Leaderboard & Arena platform for evaluating and comparing AI image generation models. The platform allows users to compare different text-to-image AI models through structured evaluation and competitive ranking systems.
