AINeutralarXiv – CS AI · Jun 116/10
🧠Researchers propose the LLM Data Auditor framework to systematically evaluate the quality and trustworthiness of synthetic data generated by large language models across six modalities. The framework shifts evaluation focus from downstream task performance to intrinsic data properties, revealing significant deficiencies in current evaluation practices and offering recommendations for improvement.
AINeutralarXiv – CS AI · Jun 116/10
🧠Researchers introduce BioDivergence, a new evaluation framework that distinguishes between genuine contradictions and context-dependent divergences in biomedical research claims. The framework includes a six-class taxonomy and 13-axis ontology to capture why studies produce seemingly conflicting results, with a released benchmark of 11,865 claim pairs showing that current NLI models struggle with contextual understanding.
AINeutralarXiv – CS AI · Jun 96/10
🧠Researchers developed the first evaluation framework for autonomous AI defense agents operating within commercial endpoint detection and response (EDR) systems, revealing critical gaps between simulation environments and real-world enterprise security. Testing with Microsoft Defender XDR and LLM-based agents uncovered that commercial EDR telemetry is optimized for human analysts rather than benchmarking, creating attribution challenges and unpredictable autonomous system behavior.
🧠 Claude🧠 Sonnet
AINeutralarXiv – CS AI · Jun 96/10
🧠Researchers introduce AVI-Bench, a comprehensive benchmark for evaluating audio-visual intelligence in multimodal large language models across perception, understanding, and reasoning tasks. The study reveals significant limitations in current models and proposes a taxonomy to guide development of more robust audio-visual AI systems.
AINeutralarXiv – CS AI · Jun 86/10
🧠Researchers propose CapCode and CapReward, frameworks designed to detect and prevent AI coding agents from achieving high evaluation scores through shortcuts rather than genuine task-solving. By capping the maximum achievable non-cheating performance below 100%, scores above the cap serve as evidence of deceptive behavior, enabling more reliable agent evaluation.
AINeutralarXiv – CS AI · Jun 86/10
🧠Researchers introduce VALUEFLOW, a comprehensive framework for aligning Large Language Models with diverse human values through hierarchical extraction, calibrated intensity evaluation, and steerable control mechanisms. The system addresses fundamental limitations in existing preference-based alignment approaches by enabling precise, multi-theory value alignment at controlled intensities across different models.
AINeutralarXiv – CS AI · Jun 56/10
🧠Researchers introduce CausalPhys, a benchmark with over 3,000 curated video and image questions designed to evaluate how well vision-language models understand causal physical reasoning. The work includes expert-annotated causal graphs and proposes Causal Rationale-informed Fine-Tuning (CRFT) to improve VLM performance on physical world reasoning tasks.
AIBullisharXiv – CS AI · Jun 46/10
🧠Researchers introduce SCORE, a self-evolving co-evolutionary framework that jointly trains evaluation and generation models for deep research report generation. The approach addresses limitations in LLM-based research agents by enabling evaluators to dynamically adapt standards as solver performance improves, demonstrating consistent quality improvements over static evaluation methods.
AINeutralarXiv – CS AI · Jun 46/10
🧠Researchers introduce NoRA, a visual reasoning benchmark that evaluates whether AI models can generate and justify appropriate actions in first-person video scenarios through explicit reasoning graphs. The benchmark reveals that current multimodal language models struggle to construct complete action spaces and properly ground decisions in visible evidence, highlighting a critical gap between selecting plausible actions and explaining them through verifiable reasoning.
AINeutralarXiv – CS AI · Jun 26/10
🧠ForeSci introduces a new benchmark for evaluating whether large language model agents can make forward-looking research decisions using only historical evidence, testing 500 tasks across AI domains. The research reveals that while explicit evidence organization improves traceability, a fundamental evidence-decision decoupling problem persists where agents cite relevant sources but reach incorrect conclusions.
AINeutralarXiv – CS AI · Jun 26/10
🧠Merkle has developed BADGER, a unified evaluation framework that combines text-to-SQL assessment with agentic behavior evaluation for enterprise AI systems. The framework achieves substantial agreement with human expert judgment (Cohen's kappa=0.717) and outperforms six competing evaluation approaches, addressing a critical gap in production-grade AI system assessment.
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers introduce AgentCL, an evaluation framework for assessing continual learning in language agents, along with MemProbe, a memory design method that helps agents accumulate and reuse knowledge across tasks while avoiding interference. The framework uses controlled task streams to rigorously measure how well agents learn and transfer knowledge over time, revealing that current memory designs struggle to balance learning plasticity with stable knowledge reuse.
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers propose a novel evaluation framework for brain-computer interfaces that independently controls the speed-accuracy trade-off through tunable parameters, separating these metrics to enable transparent, application-specific optimization without modifying the underlying classifier.
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers introduce AblationBench, a benchmark suite for evaluating language model agents on ablation planning tasks in AI research. The study finds that frontier LMs achieve only 45% accuracy on average, significantly below human performance, highlighting challenges in automating scientific research methodologies.
🏢 Hugging Face
AINeutralarXiv – CS AI · Jun 16/10
🧠Researchers propose a persona-based evaluation framework that replaces traditional monolithic AI benchmarking with diverse synthetic cognitive profiles to better capture cultural and demographic variability in human judgment. While generative models can instantiate these personas consistently, the study reveals systematic degradation in persona coherence over time, suggesting static alignment approaches are insufficient and dynamic regulatory mechanisms are needed.
AINeutralarXiv – CS AI · Jun 16/10
🧠OpenSTBench introduces a unified evaluation framework for assessing speech translation systems across multiple dimensions including translation quality, speech quality, speaker preservation, and temporal consistency. The framework addresses a critical gap in the field by enabling comprehensive comparison of heterogeneous speech translation outputs that differ in modality and timing behavior, with code and datasets made publicly available.
AINeutralarXiv – CS AI · Jun 16/10
🧠Researchers introduce SURE, a unified experimentation framework that standardizes evaluation metrics and training pipelines for speech understanding models, addressing reproducibility challenges that have hindered fair comparison of speech foundation models and Speech LLMs across different deployment scenarios.
AINeutralarXiv – CS AI · Jun 16/10
🧠SPECTRA is a new framework for generating synthetic text corpora and retrieval test collections at scale, enabling researchers to stress-test information retrieval systems without expensive human annotation. The system can produce corpora up to 60,000 documents while maintaining controllable vocabulary distributions and deterministic relevance labels, serving as a diagnostic complement to traditional evaluation methods.
AIBullisharXiv – CS AI · May 296/10
🧠Researchers introduce DynSess, a framework that evaluates and optimizes role-playing agents at the session level rather than individual turns, enabling LLMs to maintain character consistency across extended conversations. The framework includes improved evaluation metrics, optimized training methods (DSPO and GSRPO), and demonstrates performance matching larger models with fewer parameters.
AINeutralarXiv – CS AI · May 296/10
🧠Researchers introduce CalArena, a large-scale benchmark for evaluating post-hoc calibration methods in machine learning, covering nearly 2000 experiments across diverse tasks and model types. The study reveals that smooth calibration functions significantly outperform binning-based approaches, and provides open-source implementations to standardize calibration research.
AINeutralarXiv – CS AI · May 296/10
🧠The BEAMS Initiative establishes benchmarks to evaluate AI tools for modeling and simulation, ensuring they complement human expertise rather than replace it. Testing reveals that current AI-enabled modeling tools excel at discussion and qualitative tasks but struggle with causal reasoning and quantitative error correction, with performance varying significantly across different LLM implementations.
AINeutralarXiv – CS AI · May 296/10
🧠Researchers introduce Cookie-Bench, a comprehensive 1,000-query web development benchmark, and Cookie-Frame, an autonomous evaluation framework that assesses LLM-generated web applications through static perception, agent-driven interaction, and dynamic scoring. The approach eliminates reliance on reference implementations while aligning closely with human expert ratings, revealing significant performance gaps across 13 frontier LLMs.
AINeutralarXiv – CS AI · May 286/10
🧠Researchers introduce Picid, a standardized evaluation infrastructure for Prognostics and Health Management (PHM) that addresses the reproducibility crisis in predictive maintenance across industries. The framework formalizes dataset construction, preprocessing, and evaluation metrics to enable fair comparisons of fault detection, diagnostics, and prognostics models across diverse domains like batteries, bearings, and engines.
🏢 Meta
AINeutralarXiv – CS AI · May 286/10
🧠Researchers propose FeasiGen, a framework for automatically generating infeasible task benchmarks to evaluate whether AI agents recognize when tasks cannot be completed with available tools. Testing across nine models reveals critical weaknesses, with agents continuing execution on impossible tasks up to 73.9% of the time, though multi-agent architectures show improved performance.
AIBearisharXiv – CS AI · May 276/10
🧠Researchers developed a bias-aware evaluation framework to detect anti-autistic ableism in large language models, using psychometrically-weighted annotations from autistic community members as ground truth. The study reveals that LLMs frequently produce harmful outputs, misclassify community language, and rely on surface-level keyword matching rather than contextual understanding of speaker identity and intent.
