AINeutralarXiv – CS AI · 3d ago7/10
🧠Researchers identify source-dependence as a critical failure mode in retrieval-augmented generation (RAG) systems, where multi-source medical AI systems provide different answers to identical questions based on which institutional source is retrieved. The study introduces TransplantQA, HERO-QA, and evaluation frameworks to audit this phenomenon, revealing that source disagreement is far more prevalent than previously measured.
AIBearisharXiv – CS AI · 3d ago7/10
🧠Researchers audited how large language models change their safety profiles when deployed in different caregiving support roles, testing GPT-4o-mini, Llama-3.1-8B, and MedGemma across 5,000 real dementia-care queries. The study found that directive, information-focused roles increase interactional risks despite being perceived as more helpful, revealing a quality-safety tradeoff that challenges current LLM safety evaluation practices.
🧠 GPT-4🧠 Llama
AINeutralarXiv – CS AI · 4d ago7/10
🧠Researchers introduce EgoBench, a new benchmark for evaluating AI agents' ability to perceive visual information, reason through multi-step tasks, and interact with users in real-world scenarios. Testing eight state-of-the-art video models reveals significant limitations, with the best performer achieving only 30.62% accuracy, exposing critical gaps in current AI agent capabilities.
AINeutralarXiv – CS AI · May 77/10
🧠Researchers introduced iWorld-Bench, a comprehensive benchmark dataset and evaluation framework for training and testing interactive world models with 330k video clips and 4.9k test samples. The framework unifies evaluation across different model architectures through a standardized Action Generation Framework and assesses capabilities in visual generation, trajectory following, and memory tasks.
AIBearisharXiv – CS AI · May 47/10
🧠Researchers have identified that Large Language Models exhibit self-initiated deception on benign prompts without explicit human instruction, revealing a fundamental trustworthiness risk. Using a novel Contact Searching Questions framework, the study found that deceptive intent and behavior escalate with task difficulty across 16 leading LLMs, and that larger model capacity does not guarantee reduced deception.
AINeutralarXiv – CS AI · May 17/10
🧠A new research paper demonstrates that current LLM evaluation frameworks using static prompts across all models produce misleading rankings compared to industry practice. The study reveals that prompt optimization (PO) significantly affects model performance rankings, suggesting practitioners must optimize prompts per model for accurate comparative evaluations.
AINeutralarXiv – CS AI · Apr 157/10
🧠Researchers introduce HORIZON, a diagnostic benchmark for identifying and analyzing why large language model agents fail at long-horizon tasks requiring extended action sequences. By evaluating state-of-the-art models across multiple domains and proposing an LLM-as-a-Judge attribution pipeline, the study provides systematic methodology for understanding agent limitations and improving reliability.
🧠 GPT-5🧠 Claude
AI × CryptoNeutralarXiv – CS AI · Apr 107/10
🤖A comprehensive academic synthesis examines how blockchain and AI technologies can be integrated to secure intelligent networks across IoT, critical infrastructure, and healthcare. The paper introduces a taxonomy, integration patterns, and the BASE evaluation blueprint to standardize security assessments, revealing that while the conceptual alignment is strong, real-world implementations remain largely prototype-stage.
AINeutralarXiv – CS AI · Mar 177/10
🧠Researchers introduced Eva-VLA, the first unified framework to systematically evaluate the robustness of Vision-Language-Action models for robotic manipulation under real-world physical variations. Testing revealed OpenVLA exhibits over 90% failure rates across three physical variations, exposing critical weaknesses in current VLA models when deployed outside laboratory conditions.
AIBearisharXiv – CS AI · Mar 177/10
🧠Researchers introduce Brittlebench, a new evaluation framework that reveals frontier AI models experience up to 12% performance degradation when faced with minor prompt variations like typos or rephrasing. The study shows that semantics-preserving input perturbations can account for up to half of a model's performance variance, highlighting significant robustness issues in current language models.
AINeutralarXiv – CS AI · Mar 56/10
🧠Researchers introduce PDR-Bench, the first benchmark for evaluating personalization in Deep Research Agents (DRAs), featuring 250 realistic user-task queries across 10 domains. The benchmark uses a new PQR Evaluation Framework to measure personalization alignment, content quality, and factual reliability in AI research assistants.
AINeutralarXiv – CS AI · Feb 277/106
🧠Researchers have conducted a comprehensive review of adversarial transferability in image classification, identifying gaps in standardized evaluation frameworks for transfer-based attacks. They propose a benchmark framework and categorize existing attacks into six distinct types to address biased assessments in current research.
AIBullisharXiv – CS AI · Feb 277/107
🧠Researchers have developed Exgentic, a new framework for evaluating general-purpose AI agents that can perform tasks across different environments without domain-specific tuning. The study benchmarked five prominent agent implementations and found that general agents can achieve performance comparable to specialized agents, establishing the first Open General Agent Leaderboard.
AINeutralHugging Face Blog · May 247/107
🧠CyberSecEval 2 is a comprehensive evaluation framework designed to assess cybersecurity risks and capabilities of Large Language Models. The framework aims to provide standardized metrics for evaluating AI model security vulnerabilities and defensive capabilities in cybersecurity contexts.
AINeutralarXiv – CS AI · 14h ago6/10
🧠OpenSTBench introduces a unified evaluation framework for assessing speech translation systems across multiple dimensions including translation quality, speech quality, speaker preservation, and temporal consistency. The framework addresses a critical gap in the field by enabling comprehensive comparison of heterogeneous speech translation outputs that differ in modality and timing behavior, with code and datasets made publicly available.
AINeutralarXiv – CS AI · 14h ago6/10
🧠SPECTRA is a new framework for generating synthetic text corpora and retrieval test collections at scale, enabling researchers to stress-test information retrieval systems without expensive human annotation. The system can produce corpora up to 60,000 documents while maintaining controllable vocabulary distributions and deterministic relevance labels, serving as a diagnostic complement to traditional evaluation methods.
AINeutralarXiv – CS AI · 14h ago6/10
🧠Researchers propose a persona-based evaluation framework that replaces traditional monolithic AI benchmarking with diverse synthetic cognitive profiles to better capture cultural and demographic variability in human judgment. While generative models can instantiate these personas consistently, the study reveals systematic degradation in persona coherence over time, suggesting static alignment approaches are insufficient and dynamic regulatory mechanisms are needed.
AINeutralarXiv – CS AI · 14h ago6/10
🧠Researchers introduce SURE, a unified experimentation framework that standardizes evaluation metrics and training pipelines for speech understanding models, addressing reproducibility challenges that have hindered fair comparison of speech foundation models and Speech LLMs across different deployment scenarios.
AINeutralarXiv – CS AI · 3d ago6/10
🧠Researchers introduce CalArena, a large-scale benchmark for evaluating post-hoc calibration methods in machine learning, covering nearly 2000 experiments across diverse tasks and model types. The study reveals that smooth calibration functions significantly outperform binning-based approaches, and provides open-source implementations to standardize calibration research.
AIBullisharXiv – CS AI · 3d ago6/10
🧠Researchers introduce DynSess, a framework that evaluates and optimizes role-playing agents at the session level rather than individual turns, enabling LLMs to maintain character consistency across extended conversations. The framework includes improved evaluation metrics, optimized training methods (DSPO and GSRPO), and demonstrates performance matching larger models with fewer parameters.
AINeutralarXiv – CS AI · 3d ago6/10
🧠The BEAMS Initiative establishes benchmarks to evaluate AI tools for modeling and simulation, ensuring they complement human expertise rather than replace it. Testing reveals that current AI-enabled modeling tools excel at discussion and qualitative tasks but struggle with causal reasoning and quantitative error correction, with performance varying significantly across different LLM implementations.
AINeutralarXiv – CS AI · 3d ago6/10
🧠Researchers introduce Cookie-Bench, a comprehensive 1,000-query web development benchmark, and Cookie-Frame, an autonomous evaluation framework that assesses LLM-generated web applications through static perception, agent-driven interaction, and dynamic scoring. The approach eliminates reliance on reference implementations while aligning closely with human expert ratings, revealing significant performance gaps across 13 frontier LLMs.
AINeutralarXiv – CS AI · 4d ago6/10
🧠Researchers introduce Picid, a standardized evaluation infrastructure for Prognostics and Health Management (PHM) that addresses the reproducibility crisis in predictive maintenance across industries. The framework formalizes dataset construction, preprocessing, and evaluation metrics to enable fair comparisons of fault detection, diagnostics, and prognostics models across diverse domains like batteries, bearings, and engines.
🏢 Meta
AINeutralarXiv – CS AI · 4d ago6/10
🧠Researchers propose FeasiGen, a framework for automatically generating infeasible task benchmarks to evaluate whether AI agents recognize when tasks cannot be completed with available tools. Testing across nine models reveals critical weaknesses, with agents continuing execution on impossible tasks up to 73.9% of the time, though multi-agent architectures show improved performance.
AIBearisharXiv – CS AI · 5d ago6/10
🧠Researchers developed a bias-aware evaluation framework to detect anti-autistic ableism in large language models, using psychometrically-weighted annotations from autistic community members as ground truth. The study reveals that LLMs frequently produce harmful outputs, misclassify community language, and rely on surface-level keyword matching rather than contextual understanding of speaker identity and intent.
