AINeutralarXiv – CS AI · Jun 257/10
🧠Researchers introduce InvestPhilBench, a comprehensive benchmark for testing large language models' ability to reconstruct and apply expert investment decision frameworks. The v0.6 release reveals that while state-of-the-art models achieve high composite scores (0.932), they exhibit significant procedural reasoning deficits (GRA scores of 0.57-0.77), indicating that fluent prose masks deeper gaps in step-by-step investment logic.
🧠 Claude
AIBullisharXiv – CS AI · Jun 257/10
🧠Researchers tested large language models against human examiners on 32,534 real UK GCSE exam responses, finding that top-performing models achieve higher agreement with examiner consensus than examiners do with each other. The results demonstrate LLMs can reliably grade subjective tasks like essays and handle complex handwritten work, suggesting viable automated marking solutions.
AIBearisharXiv – CS AI · Jun 197/10
🧠A peer-reviewed study finds that psychological profiles assigned to large language models through human-designed tests are largely measurement artifacts rather than genuine model traits. The research, analyzing 56 instruction-tuned LLMs, reveals that directional response bias—not actual personality—drives 81-90% of differences between models, undermining the validity of using standard psychological instruments to assess LLM safety, usability, and research applications.
AIBearisharXiv – CS AI · Jun 107/10
🧠Researchers developed the first psychometric instrument designed specifically for LLMs based on their actual behavioral patterns, but found that LLMs' self-reported personality traits show virtually no correlation with their observed behavior—a critical finding for AI alignment and applications using LLMs as evaluators.
AIBearisharXiv – CS AI · Jun 107/10
🧠A new benchmarking study challenges the widespread narrative that large language models perform at expert-level on knowledge work tasks. By measuring variance and error magnitude alongside accuracy, researchers found that human experts outperformed frontier LLMs on a data analysis coding task, demonstrating that standard benchmarks fail to capture reliability and consistency—critical factors for high-stakes applications.
AIBearisharXiv – CS AI · Jun 97/10
🧠A comprehensive evaluation of 9 open-source coding LLMs across 2,707 LeetCode problems in 12 programming languages reveals significant performance gaps compared to human developers. The best model achieves only 23.64% correctness versus a 57.2% human baseline, with performance varying substantially across languages and problem types, indicating that aggregate benchmarks mask critical weaknesses in code generation systems.
AINeutralarXiv – CS AI · Jun 87/10
🧠Researchers introduced MMBU, the largest biomedical vision-language benchmark covering 35 medical imaging modalities with structured metadata. Testing 15 open-weight and 2 frontier VLMs revealed that while medical adaptation helps some models, high reported accuracy on existing benchmarks masks significant deficiencies in visual perception and domain generalization.
AIBullisharXiv – CS AI · Jun 57/10
🧠Researchers introduce Benchmark Agent, an autonomous AI system that automates the creation of machine learning benchmarks to address labor-intensive construction and performance saturation issues. The framework successfully generated 15 diverse benchmarks across text and multimodal understanding tasks, demonstrating that continually evolving benchmarks can accelerate LLM and MLLM development with minimal human oversight.
AIBullisharXiv – CS AI · Jun 37/10
🧠TriEval introduces an open-source pipeline for evaluating large language models across bias, toxicity, and truthfulness simultaneously while requiring minimal computational resources. The tool runs on standard laptops without GPU clusters, making rigorous LLM safety testing accessible to researchers with limited budgets, and reveals significant performance differences between open-source and closed-source models.
🧠 Claude🧠 Llama
AIBearisharXiv – CS AI · Jun 27/10
🧠Researchers introduce ClinEnv, an interactive benchmark that evaluates large language models as attending physicians making real clinical decisions across multiple stages of patient care. The study reveals that even the strongest models achieve only 0.31 decision F1 scores, with significant gaps between diagnostic accuracy and clinical management quality, exposing how outcome-focused evaluations mask deficiencies in information-gathering processes.
AIBullisharXiv – CS AI · May 287/10
🧠Researchers present a unified evaluation framework for assessing LLM agentic capabilities, integrating 7 benchmarks across 24 domains with standardized testing methodology. The framework disentangles intrinsic model performance from implementation artifacts, revealing that scaffold choices and environmental volatility significantly impact benchmark results across 15 models tested.
🏢 Meta🏢 Hugging Face
AIBearisharXiv – CS AI · May 277/10
🧠Researchers introduced LiveK12Bench, a dynamic benchmark for evaluating Large Multimodal Models on realistic high school examinations across multiple disciplines. The study reveals that advanced LMMs like GPT-4 experience significant performance degradation when subjected to exam-realistic constraints, dropping from 79 to 53 points when process rigor and efficiency are jointly evaluated, exposing critical gaps between theoretical capabilities and practical educational readiness.
🧠 GPT-5
AIBearisharXiv – CS AI · May 277/10
🧠Researchers reveal that AI models can possess stable factual knowledge while failing dramatically at compositional reasoning—assembling facts into logical chains—a problem invisible to standard benchmark metrics. The study introduces a diagnostic protocol showing post-training improvements mask directional shifts in composition capability, with failures often rooted in generation-time constraints rather than fundamental model limitations.
AINeutralarXiv – CS AI · May 277/10
🧠Researchers introduce BeQu, a new benchmark that evaluates LLM knowledge through open-ended prompts rather than predefined questions, addressing availability bias in existing benchmarks. The paradigm shift from narrow question-answering to characterizing naturally expressed knowledge provides deeper insights into parametric knowledge across 10,000 entities and multiple language models.
AINeutralarXiv – CS AI · May 117/10
🧠Researchers introduce a scenario-grounded benchmark for evaluating large language models in scientific discovery, revealing significant performance gaps compared to general science benchmarks. The framework tests LLMs across biology, chemistry, materials, and physics through project-level tasks involving hypothesis generation and experimental design, showing that current models remain distant from achieving general scientific superintelligence despite demonstrating promise in specific applications.
AINeutralarXiv – CS AI · Apr 157/10
🧠Researchers propose a cognitive diagnostic framework that evaluates large language models across fine-grained ability dimensions rather than aggregate scores, enabling targeted model improvement and task-specific selection. The approach uses multidimensional Item Response Theory to estimate abilities across 35 dimensions for mathematics and generalizes to physics, chemistry, and computer science with strong predictive accuracy.
AINeutralarXiv – CS AI · Apr 147/10
🧠Researchers introduce General365, a benchmark revealing that leading LLMs achieve only 62.8% accuracy on general reasoning tasks despite excelling in domain-specific domains. The findings highlight a critical gap: current AI models rely heavily on specialized knowledge rather than developing robust, transferable reasoning capabilities applicable to real-world scenarios.
AIBearisharXiv – CS AI · Apr 107/10
🧠Researchers reveal that Large Language Models exhibit self-preference bias when evaluating other LLMs, systematically favoring outputs from themselves or related models even when using objective rubric-based criteria. The bias can reach 50% on objective benchmarks and 10-point score differences on subjective medical benchmarks, potentially distorting model rankings and hindering AI development.
AINeutralarXiv – CS AI · Mar 177/10
🧠Researchers introduce AVA-Bench, a new benchmark that evaluates vision foundation models (VFMs) by testing 14 distinct atomic visual abilities like localization and depth estimation. This approach provides more precise assessment than traditional VQA benchmarks and reveals that smaller 0.5B language models can evaluate VFMs as effectively as 7B models while using 8x fewer GPU resources.
AIBearisharXiv – CS AI · Jun 236/10
🧠Researchers introduce Coherence Under Commitment (CUC), a new evaluation framework that exposes a critical flaw in LLM logical reasoning: models can achieve coherence by refusing to make decisions rather than reasoning correctly. Testing on small language models reveals a stark trade-off where more decisive models contradict themselves frequently, while conservative models abstain from answering.
AINeutralarXiv – CS AI · Jun 196/10
🧠Researchers introduce QMFOL, an automated framework for generating controlled-complexity logical reasoning benchmarks to evaluate large language models. The resulting QMFOLBench dataset of 2,880 instances reveals that LLM reasoning performance degrades significantly with increased logical complexity, with models showing consistent bias toward true-labeled tasks over false or unknown ones.
AINeutralarXiv – CS AI · Jun 116/10
🧠Researchers introduce RAIL, a new evaluation framework for large audio-language models grounded in cognitive science principles rather than task-specific metrics. The benchmark, based on the Cattell-Horn-Carroll cognitive framework, reveals that state-of-the-art audio-language models exhibit uneven performance across core auditory cognitive abilities, highlighting a gap between how humans and current AI systems process audio information.
AINeutralarXiv – CS AI · Jun 106/10
🧠Researchers introduce V-REX, a new evaluation benchmark for vision-language models that assesses their ability to perform complex, multi-step visual reasoning through Chain-of-Questions (CoQ) methodology. The framework disentangles VLMs' planning and information-gathering capabilities, revealing significant performance gaps and substantial room for improvement in exploratory visual reasoning tasks.
AINeutralarXiv – CS AI · Jun 96/10
🧠Researchers introduce TheoremBench, a comprehensive Lean4 benchmark for evaluating large language models on formal mathematics theorem proving. Unlike existing competition-focused benchmarks, TheoremBench assesses how LLMs handle longer, dependency-rich mathematical proofs through both standalone theorems and structured families of related subtasks, revealing that current models remain inefficient and biased toward simpler problems.
AINeutralarXiv – CS AI · Jun 96/10
🧠Researchers introduce ClinicalBr, the first bilingual clinical benchmark using 2,892 real Brazilian Portuguese-English case reports to evaluate large language models. The study reveals that English-language advantages in clinical AI are task-dependent, with Portuguese performing comparably in differential diagnosis, exam recommendations, and treatment planning.
