AIBullisharXiv – CS AI · 16h ago7/10
🧠Researchers demonstrate that efficient LLM benchmarking can be substantially improved by treating it as a multiple regression problem with kernel ridge regression and applying minimum redundancy maximum relevance (mRMR) feature selection. The approach achieves lower prediction errors and faster computation than existing methods while maintaining consistency across different data splits.
AINeutralarXiv – CS AI · 3d ago7/10
🧠Researchers propose Critique-Resilient Benchmarking, a new framework for evaluating large language models when human comprehension of tasks becomes infeasible. The method uses adversarial evaluation where answers are deemed correct if no convincing counterargument exists, allowing meaningful comparison of frontier LLMs even as they saturate traditional benchmarks.
AIBearisharXiv – CS AI · 4d ago7/10
🧠Researchers introduce PortBench, a comprehensive benchmark for evaluating large language models in portfolio management tasks. The study reveals that 90% of tested LLMs fail to outperform basic equal-weight allocation strategies, highlighting significant gaps between LLM performance on financial QA tasks and real-world portfolio decision-making.
AIBullisharXiv – CS AI · 4d ago7/10
🧠Researchers introduce RAGe, a benchmarking framework designed to optimize Retrieval-Augmented Generation (RAG) applications by evaluating trade-offs between accuracy, efficiency, and scalability. The framework enables developers to identify optimal pipeline components for domain-specific datasets while accounting for hardware constraints, making RAG development more accessible on consumer-grade hardware.
AINeutralarXiv – CS AI · 5d ago7/10
🧠Researchers have identified significant measurement bias in production LLM benchmarking tools, where single-process architectures and Python's Global Interpreter Lock artificially inflate latency metrics at scale. The study proposes a multi-process evaluation framework and a new normalized metric (NTPOT) to accurately measure LLM serving performance under production-level concurrency.
AINeutralarXiv – CS AI · May 97/10
🧠Researchers introduce a framework for evaluating whether AI creative systems cause population-level diversity collapse, where individual output quality improves while collective idea similarity increases. Testing three frontier LLMs across creative tasks, the study finds they fall below diversity parity with humans and proposes design interventions to mitigate crowding effects at development time.
AINeutralarXiv – CS AI · Apr 147/10
🧠Researchers introduce General365, a benchmark revealing that leading LLMs achieve only 62.8% accuracy on general reasoning tasks despite excelling in domain-specific domains. The findings highlight a critical gap: current AI models rely heavily on specialized knowledge rather than developing robust, transferable reasoning capabilities applicable to real-world scenarios.
AINeutralarXiv – CS AI · 3d ago6/10
🧠Researchers introduce LogDx-CI, a benchmark comparing 11 log-reduction tools for debugging CI failures using LLMs, finding that hybrid grep+tail routers achieve the best cost-quality tradeoff while agent-loop systems can recover from weak contexts through iterative tool calls, though at higher computational cost.
🏢 OpenAI🧠 GPT-5🧠 Claude
AINeutralarXiv – CS AI · 3d ago6/10
🧠Researchers introduce Code-QA-Bench, an automated framework that generates repository-level code understanding benchmarks while distinguishing genuine code comprehension from documentation recall. Testing four frontier AI models reveals that code access is the primary driver of performance, while documentation provides marginal benefits, suggesting current models excel at code reasoning when source material is available.
AINeutralarXiv – CS AI · 4d ago6/10
🧠Researchers introduce BenGER, a comprehensive benchmark dataset for evaluating large language models on German legal reasoning tasks, comprising 596 exam-style cases and 531 doctrinal reasoning problems. The study demonstrates that LLM-as-a-Judge frameworks can achieve near-human consistency in legal assessment, with human-AI collaboration substantially outperforming unaided human performance.
AINeutralarXiv – CS AI · 5d ago6/10
🧠Researchers introduce MeDial-Speech, a new 111+ hour speech dataset for training medical AI systems to conduct patient consultations across four health conditions. The study benchmarks state-of-the-art LLMs including Claude Sonnet 4, GPT-5 mini, and DeepSeek-V3, revealing that while Claude Sonnet 4 achieves 71-75% accuracy in medical dialogue tasks, all models exhibit significant overconfidence in their probabilistic predictions.
🏢 Hugging Face🧠 GPT-5🧠 Claude
AINeutralarXiv – CS AI · 5d ago6/10
🧠Researchers introduced FrontierOR, a benchmark that tests whether leading LLMs can design efficient optimization algorithms for real-world large-scale problems. The evaluation of seven models reveals significant limitations: even frontier models outperform Gurobi (a standard solver) in only 31% of cases, highlighting a substantial gap between LLM capabilities in formulation and practical algorithmic optimization.
AINeutralarXiv – CS AI · May 126/10
🧠Researchers introduce VeriContest, a benchmark of 946 competitive-programming problems designed to evaluate AI models' ability to generate not just functional code but also formal specifications and machine-checkable proofs. Testing ten state-of-the-art models reveals a dramatic capability gap: while the strongest model achieves 92% accuracy on code generation alone, performance plummets to 48% on specifications, 14% on proofs, and just 5% end-to-end, identifying proof generation as the critical bottleneck for verifiable code generation systems.
AINeutralarXiv – CS AI · May 116/10
🧠Researchers introduce PostEDA-Bench, a hierarchical benchmark for evaluating LLM-based agents in Electronic Design Automation tasks, specifically targeting Design Rule Check (DRC) fixing and Power-Performance-Area (PPA) optimization. Testing eight LLMs across 145 tasks reveals significant performance gaps, with best success rates of 36.66% for complex DRC reasoning and only 20% for multi-objective PPA optimization, indicating substantial room for improvement in AI-assisted chip design automation.
AINeutralarXiv – CS AI · May 115/10
🧠ENGINEERING Ingegneria Informatica has released EngGPT2MoE-16B-A3B, a 16-billion parameter Mixture of Experts language model that demonstrates competitive or superior performance compared to Italian and international open-source LLMs across multiple benchmarks. The model represents a notable advancement for Italian-language AI capabilities while positioning itself competitively within the global open-source LLM landscape.
🧠 GPT-5🧠 Llama
AINeutralarXiv – CS AI · May 116/10
🧠Researchers introduce TEA-Bench, the first interactive benchmark for evaluating how external tools improve emotional support conversation (ESC) systems. Testing nine LLMs reveals that tool augmentation reduces hallucination and improves support quality, but effectiveness depends heavily on model capacity—stronger models leverage tools more effectively than weaker ones.
AIBullisharXiv – CS AI · May 76/10
🧠Gosset, a curated AI platform for pharmaceutical asset discovery, outperforms leading frontier LLMs (Claude, GPT-5.5, Gemini, Perplexity) by 3.2x on drug discovery queries, achieving perfect precision and complete recall on niche oncology and immunology targets. The research demonstrates that specialized, annotated databases significantly outperform general-purpose models with web search for domain-specific tasks.
🏢 Perplexity🧠 GPT-5🧠 Claude
AINeutralarXiv – CS AI · May 46/10
🧠Researchers introduce ArabCulture-Dialogue, a new dataset for evaluating large language models' cultural reasoning across 13 Arabic-speaking countries in both Modern Standard Arabic and regional dialects. Benchmarking reveals significant performance gaps, with LLMs consistently underperforming on dialectal Arabic compared to standardized variants, highlighting a critical blind spot in AI language model training.
AINeutralarXiv – CS AI · May 46/10
🧠Researchers introduce MemoryBench, a new benchmark for evaluating how large language models learn and improve from accumulated user feedback over time. The framework addresses limitations in existing memory benchmarks by testing continual learning across multiple domains and languages, revealing that current state-of-the-art systems perform poorly on these tasks.
AIBullisharXiv – CS AI · Apr 206/10
🧠Researchers have introduced VLegal-Bench, the first comprehensive benchmark for evaluating large language models on Vietnamese legal tasks, comprising 10,450 expert-annotated samples grounded in real legal documents. The benchmark uses Bloom's cognitive taxonomy to assess LLM performance across practical legal scenarios, establishing a standardized framework for developing more reliable AI-assisted legal systems in Vietnam.
AINeutralarXiv – CS AI · Apr 136/10
🧠Researchers introduced NLCO, a benchmark for evaluating large language models on natural-language combinatorial optimization problems without external solvers or code generation. Testing across modern LLMs reveals that while high-performing models handle small instances well, performance degrades significantly as problem complexity increases, with graph-structured and bottleneck-objective problems proving particularly challenging.
AIBullisharXiv – CS AI · Mar 37/108
🧠Researchers have introduced LitBench, a new benchmarking tool designed to develop and evaluate domain-specific large language models for literature-related tasks. The tool uses graph-centric data curation to generate domain-specific literature sub-graphs and creates training datasets, with results showing small domain-specific LLMs achieving competitive performance against state-of-the-art models like GPT-4o.
AINeutralarXiv – CS AI · Mar 36/104
🧠Researchers introduced EHR-ChatQA, a new benchmark for testing AI agents that interact with Electronic Health Record databases through natural language queries. The benchmark reveals significant reliability gaps in current state-of-the-art LLMs, with success rates dropping substantially when consistency across multiple trials is required.
