AIBullisharXiv – CS AI · Jun 86/10
🧠Researchers introduce CoQuIR, a comprehensive benchmark for evaluating code retrieval systems across quality dimensions including correctness, efficiency, security, and maintainability. Testing 23 retrieval models reveals that even top performers struggle to distinguish high-quality code from buggy or insecure alternatives, with preliminary training methods showing promise in improving quality-awareness without sacrificing semantic relevance.
AINeutralarXiv – CS AI · Jun 56/10
🧠Researchers introduce Entropy-Based Evaluation of AI Agents (EEA), a lightweight framework that measures AI agent behavior through entropy metrics rather than relying solely on task completion rates. The framework introduces six new metrics including action entropy, trajectory entropy, and exploration efficiency, with Python implementation designed for integration with popular agent frameworks like LangChain.
AINeutralarXiv – CS AI · Jun 56/10
🧠A research paper demonstrates that parameter-efficient fine-tuning of small language models (3B parameters) using LoRA achieves competitive performance for telecommunications customer support while consuming significantly less energy than larger models. Critically, the study reveals that traditional validation loss metrics poorly predict real-world conversational quality, with the lowest-loss model ranking 6th-7th in human-aligned evaluation while the worst-loss model ranked first.
🧠 GPT-5🧠 Claude🧠 Gemini
AIBullisharXiv – CS AI · Jun 46/10
🧠MM-BizRAG introduces a structured approach to multimodal retrieval-augmented generation for enterprise document analysis, dynamically routing documents through layout-specific processing pipelines and outperforming existing vision-centric baselines by up to 32% on heterogeneous enterprise datasets. The system decouples retrieval from generation contexts and introduces FastRAGEval, a cost-efficient evaluation metric for RAG system quality assessment.
AINeutralarXiv – CS AI · Jun 46/10
🧠Researchers introduce QO-Bench, a diagnostic benchmark for evaluating retrieval-augmented generation (RAG) systems on structured database-style queries over text. The benchmark reveals that current RAG systems excel at finding relevant passages but fail to preserve typed values needed for query operators like joins and counting, identifying operator execution rather than retrieval as the core bottleneck.
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers introduce CV-Arena, a benchmark containing 12,000 high-resolution image instruction pairs to evaluate how well AI systems solve professional-grade computer vision tasks. The study proposes Active Elo, a human-AI collaborative evaluation protocol, and reveals that current models struggle with instruction adherence, physical reasoning, and detail preservation in real-world editing workflows.
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers propose a new evaluation framework for audio-driven talking head generation that uses sequence-level alignment instead of frame-by-frame comparison. The method accounts for natural timing variations in speech-driven facial motion, providing more accurate assessment of generative model quality across different datasets and speaking styles.
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers identify a fundamental mismatch between pairwise ranking metrics (AP and FPR-95) commonly used to evaluate multi-view object association models and the actual one-to-one assignment objective these systems aim to solve. The study demonstrates that optimal ranking performance does not guarantee correct assignments, and proposes Sinkhorn-based normalization as a solution to better align evaluation metrics with real-world performance goals.
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers introduce CityTrajBench, a unified benchmark framework for evaluating vehicle trajectory generation models across urban environments. The framework standardizes datasets, preprocessing, and evaluation metrics to enable fair comparison of statistical, VAE, GAN, diffusion, and flow-matching models, revealing that no single approach dominates all quality criteria.
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers introduced LocalSearchBench, a comprehensive benchmark for testing AI agents in local life services, revealing significant performance gaps even among state-of-the-art large reasoning models. The benchmark comprises 1.3M merchant entries and 900 multi-hop reasoning tasks, exposing critical weaknesses in completeness and faithfulness that underscore the need for domain-specific AI agent development.
AINeutralarXiv – CS AI · Jun 16/10
🧠Researchers propose a unified framework for improving Panoptic Quality (PQ) metric evaluation in image segmentation by recasting segment matching as a constrained bipartite assignment problem. The framework systematically explores multiple matching strategies below IoU 0.5 threshold and extends to part-aware segmentation evaluation, with an open-source implementation released.
AINeutralarXiv – CS AI · May 296/10
🧠Researchers have developed InsightEval, a new benchmark for evaluating how well AI agents discover insights from large datasets. The work addresses critical flaws in the existing InsightBench framework, including format inconsistencies and redundant insights, and introduces a novel metric to measure exploratory performance in LLM-driven data analysis systems.
AINeutralarXiv – CS AI · May 296/10
🧠Researchers introduce MPDocBench-Parse, a new benchmark dataset for evaluating multi-page document parsing systems across realistic, complex scenarios. The benchmark comprises 433 manually annotated documents spanning 3,246 pages in 15 document types, revealing that existing AI models excel at basic text extraction but struggle with semantic continuity, visual content preservation, and hierarchical structure recovery.
AINeutralarXiv – CS AI · May 286/10
🧠Researchers present a systematic evaluation of large language models' reasoning capabilities on Boolean satisfiability problems, introducing a paired-formula protocol with Accurate Differentiation Rate (ADR) metric that reveals conventional accuracy metrics can be misleading, as models often succeed through heuristics rather than genuine reasoning.
AINeutralarXiv – CS AI · May 286/10
🧠Researchers introduce Contextual Alternative Choice (CAC), a new evaluation method that measures both syntactic and functional properties of language models using metrics derived from child language acquisition studies. While some large language models approach human-level performance on these benchmarks, none trained on comparable data volumes simultaneously meet both formal and functional standards that children achieve early in development.
AINeutralarXiv – CS AI · May 286/10
🧠Researchers challenge the widespread practice of using global token perplexity to evaluate generative spoken language models, arguing this metric fails to account for fundamental differences between speech and text modalities. The study proposes alternative likelihood- and generative-based evaluation methods that correlate more strongly with human perception, revealing that performance gaps between leading models and human baselines are smaller than previously believed.
🏢 Perplexity
AINeutralarXiv – CS AI · May 276/10
🧠Researchers propose π-Soft-NC and π-Soft-NS, improved evaluation metrics for assessing input attribution methods in large language models that control for the number of retained words, addressing a fundamental bias in existing faithfulness evaluation frameworks. They also introduce Grad-ELLM, a gradient-based attribution method designed for decoder-only LLMs that combines gradient and attention mechanisms for stronger explanatory performance.
🧠 Llama
AINeutralarXiv – CS AI · May 276/10
🧠Researchers introduce GICDM, an improved method for evaluating generative models that corrects the hubness phenomenon—a distortion in high-dimensional spaces that skews distance-based metrics and nearest-neighbor relationships. The technique builds on classical ICDM and includes multi-scale extensions, demonstrating improved alignment with human assessment across synthetic and real benchmarks.
AINeutralarXiv – CS AI · May 126/10
🧠The CODS 2025 AssetOpsBench competition retrospective reveals critical gaps between public and private evaluation metrics in multi-agent orchestration systems. Hidden test sets dramatically altered performance rankings, particularly in execution tasks where correlations turned negative, while successful teams prioritized guardrails over novel architectures.
AINeutralarXiv – CS AI · May 126/10
🧠Researchers introduce VT-Bench, the first comprehensive benchmark for visual-tabular multi-modal learning, aggregating 14 datasets with 756K samples across 9 domains. The benchmark evaluates 23 models and reveals significant gaps in current approaches for combining image and tabular data, particularly in high-stakes sectors like healthcare.
AINeutralarXiv – CS AI · May 116/10
🧠Researchers propose that coding agents need to move beyond autonomy toward proactivity—the ability to anticipate developer needs, connect signals across tools, and make unsolicited but valuable interventions. The work introduces a taxonomy of proactivity levels and evaluation metrics (Insight Decision Quality, Context Grounding Score, Learning Lift) to measure whether agent behavior genuinely improves development workflows rather than merely increasing activity.
AINeutralarXiv – CS AI · May 116/10
🧠Researchers introduce TRACE, a benchmark dataset for evaluating tourism recommendation systems that combine multi-turn dialogue, verifiable review citations, and rejection recovery. The dataset reveals a significant gap in existing conversational recommender systems: LLMs excel at recall but cite weakly, while retrieval-based systems ground better but struggle with accuracy and adaptation.
AIBullisharXiv – CS AI · May 116/10
🧠Researchers demonstrate that automated evaluation metrics can reliably assess AI-generated responses to patient hospitalization questions, matching human expert ratings across 2,800 responses from 28 AI systems. This approach addresses the scalability limitations of manual expert review while maintaining accuracy across three key dimensions: question answering, clinical evidence use, and medical knowledge application.
AIBullisharXiv – CS AI · Apr 136/10
🧠Researchers propose Interactive ASR, a new framework that combines semantic-aware evaluation using LLM-as-a-Judge with multi-turn interactive correction to improve automatic speech recognition beyond traditional word error rate metrics. The approach simulates human-like interaction, enabling iterative refinement of recognition outputs across English, Chinese, and code-switching datasets.
AIBullisharXiv – CS AI · Mar 166/10
🧠Researchers introduce a formal planning framework that maps LLM-based web agents to traditional search algorithms, enabling better diagnosis of failures in autonomous web tasks. The study compares different agent architectures using novel evaluation metrics and a dataset of 794 human-labeled trajectories from WebArena benchmark.
