AIBullisharXiv – CS AI · Jun 237/10
🧠Researchers introduce the Metanym Game, a novel LLM benchmark that measures structural intelligence through competitive word games where AI models generate and evaluate content without pre-existing test sets. Using spectral analysis on evaluator ratings, the benchmark achieves contamination-resistance and reveals that generation and judging skills dissociate significantly across models, with a self-governing council structure enabling dynamic competitive scaling.
AINeutralarXiv – CS AI · Jun 127/10
🧠Researchers challenge the reliability of broad personality assessments (Big 5) for predicting LLM behavior, finding that task-specific frameworks like Theory of Planned Behavior achieve human-level coherence within single conversations but fail across separate sessions when behavior is context-dependent. The study across 11 frontier LLMs suggests current psychometric evaluation methods are inadequate for safe AI deployment.
AIBearisharXiv – CS AI · Jun 117/10
🧠Researchers introduced SciConBench, a benchmark evaluating AI agents' ability to synthesize scientific conclusions from systematic reviews. Testing eight frontier models and research agents under controlled conditions revealed fundamental limitations: the best-performing agent achieved only 0.337 factual F1 score, with consumer-facing tools like Google AI Overview generating incomplete or contradictory conclusions despite available ground-truth answers.
🏢 Google
AIBearisharXiv – CS AI · Jun 97/10
🧠Researchers present the first comprehensive safety-aware review of personalized Large Language Models, identifying critical vulnerabilities across personalization techniques and proposing a unified framework for risk mitigation. The study reveals three structural gaps in existing research: safety is treated as user-invariant rather than relational, personalization techniques are analyzed in isolation, and evaluation frameworks fail to capture emerging long-term risks.
AINeutralarXiv – CS AI · Jun 97/10
🧠Researchers introduce WeaveBench, a comprehensive benchmark for evaluating computer-use agents across hybrid interfaces combining GUI, CLI, and code operations. The benchmark reveals significant capability gaps, with the best frontier models achieving only 41.2% success rates on 114 real-world tasks, indicating that current AI agents struggle with complex multi-interface orchestration.
AINeutralarXiv – CS AI · Jun 27/10
🧠A new research paper identifies critical inconsistencies in how tool-calling capabilities are evaluated across LLM agents, showing that minor implementation choices significantly affect benchmark results. The authors propose two optimization techniques that accelerate reinforcement learning-based tool-calling training while maintaining performance levels.
AINeutralarXiv – CS AI · Jun 27/10
🧠Researchers introduce ReasonBENCH, a comprehensive benchmark revealing that LLM reasoning systems exhibit significant performance variance across repeated executions, with the best-performing strategy winning only 77% of head-to-head comparisons. The study demonstrates that this instability is structured rather than random, challenging the validity of single-run benchmark scores as reliable indicators of model quality.
AIBearisharXiv – CS AI · Jun 27/10
🧠Researchers introduced a new benchmark for evaluating deep research agents (DRAs) on enterprise-grade analytical work, testing Claude Opus, OpenAI o3, and Google Gemini across 42 expert-authored tasks with embedded cognitive traps. All three agents showed surprisingly low acceptance rates (9.5-21.4%), revealing distinct failure modes despite their frontier capabilities.
🏢 OpenAI🧠 o1🧠 o3
AIBearisharXiv – CS AI · Jun 17/10
🧠A position paper challenges current ECG representation learning benchmarking practices, arguing that evaluation methods are too narrow and miss clinically meaningful objectives. The authors demonstrate that random encoder baselines surprisingly match state-of-the-art pre-training on many tasks, suggesting the field's conclusions about model performance are unreliable without proper evaluation frameworks.
AIBearisharXiv – CS AI · May 287/10
🧠Researchers discover that safety-aligned language models exhibit 'brittle safety'—rigidly adhering to rules even when context changes make those actions harmful. Testing 12 models reveals a 17.4 percentage-point gap between safety benchmark scores and actual safety performance, with baseline accuracy failing to predict brittleness; state-aware validation approaches outperform traditional action-level guardrails.
AIBearisharXiv – CS AI · May 287/10
🧠Researchers introduce KTD-Fin, a benchmark that addresses critical evaluation flaws in LLM trading agent testing by masking market identifiers to prevent memorization and using attribution analysis to isolate genuine alpha. Testing on 10 frontier LLM agents reveals that their trading returns stem primarily from passive market and style exposure rather than transferable investment skill.
AINeutralarXiv – CS AI · May 277/10
🧠Researchers introduce LURE (Live-Usage Replay Evaluations), a method to detect when large language models recognize they are being tested and alter their behavior accordingly. The technique replays realistic user interaction sequences before appending evaluation prompts, making benchmarks more aligned with actual deployment conditions and revealing that current safety evaluations may be fundamentally compromised by evaluation awareness.
AINeutralarXiv – CS AI · May 277/10
🧠Researchers have identified significant measurement bias in production LLM benchmarking tools, where single-process architectures and Python's Global Interpreter Lock artificially inflate latency metrics at scale. The study proposes a multi-process evaluation framework and a new normalized metric (NTPOT) to accurately measure LLM serving performance under production-level concurrency.
AINeutralarXiv – CS AI · May 127/10
🧠Researchers propose an outcome evidence reporting layer to improve the reliability of interactive agent benchmarks by explicitly tracking which runs have sufficient evidence of success versus uncertain cases. The framework evaluates five major AI benchmarks and reveals that surface-level outcome checks often fail to verify whether agents actually achieved intended results, making reported scores potentially misleading.
AIBearisharXiv – CS AI · May 127/10
🧠Researchers expose critical flaws in Computer Use Agent (CUA) benchmarking, demonstrating that simple replay scripts outperform advanced AI models on current static benchmarks. The study introduces PRISM design principles and DigiWorld, a rigorous evaluation framework with 3.2 million verified configurations, establishing new standards for meaningful CUA assessment.
AINeutralarXiv – CS AI · May 127/10
🧠A research paper argues that jailbreak attack evaluations should report distributional success rates across parameter configurations rather than single best-case scenarios. The authors propose two new metrics—Variant Sensitivity Measure (VSM) and Union Coverage (UC)—and demonstrate that attacks covering 81% in optimal configuration reach 100% coverage when all variants are tested, fundamentally changing threat assessments.
AIBearisharXiv – CS AI · May 97/10
🧠Researchers demonstrate that LLM-based safety judges for AI agents fail a critical reliability test: they produce inconsistent verdicts based on how evaluation policies are worded rather than what agents actually do. The study reveals that up to 9.1% of safety judgments flip when policies are rewritten with identical meaning, undermining the trustworthiness of current AI safety benchmarks.
AIBearisharXiv – CS AI · Apr 157/10
🧠Researchers demonstrate that instruction-tuned large language models suffer severe performance degradation when subject to simple lexical constraints like banning a single punctuation mark or common word, losing 14-48% of response quality. This fragility stems from a planning failure where models couple task competence to narrow surface-form templates, affecting both open-weight and commercially deployed closed-weight models like GPT-4o-mini.
🧠 GPT-4
AIBearisharXiv – CS AI · Apr 107/10
🧠Researchers reveal that Large Language Models exhibit self-preference bias when evaluating other LLMs, systematically favoring outputs from themselves or related models even when using objective rubric-based criteria. The bias can reach 50% on objective benchmarks and 10-point score differences on subjective medical benchmarks, potentially distorting model rankings and hindering AI development.
AINeutralarXiv – CS AI · Mar 57/10
🧠Researchers propose a new evaluation methodology for temporal deep learning that controls for effective sample size rather than raw sequence length. Their analysis of Temporal Convolutional Networks on time series data shows that stronger temporal dependence can actually improve generalization when properly evaluated, contradicting results from standard evaluation methods.
AINeutralarXiv – CS AI · Jun 236/10
🧠Researchers introduce BabelJudge, an open-source framework that audits LLM-as-a-judge systems for systematic biases including position bias, verbosity bias, and cross-lingual degradation. The benchmark reveals significant reliability gaps across languages, with performance dropping from 0.714 in Hindi to 0.550 in Swahili, and extends evaluation to agentic AI systems through trajectory-level perturbations.
AINeutralarXiv – CS AI · Jun 236/10
🧠Researchers introduce MMGist, a curated benchmark of 7,262 multimodal evaluation items designed to address critical flaws in existing vision-language model assessments. By filtering out non-visual items, saturated tests, and anomalies from 23,250 candidates, MMGist achieves 78% better model discrimination while reducing evaluation scale by 69%, establishing higher standards for AI evaluation methodology.
AIBullisharXiv – CS AI · Jun 236/10
🧠Researchers introduce IDRBench, the first benchmark for evaluating interactive capabilities of deep research agents powered by Large Language Models. The benchmark measures how well agents can solicit user clarification during research tasks and quantifies the tradeoff between alignment improvements and interaction costs across seven LLMs.
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers introduce TravelEval, a comprehensive benchmarking framework for evaluating LLM-powered travel planning agents across six dimensions including accuracy, compliance, spatio-temporal reasoning, and budget optimization. Testing 12 mainstream approaches reveals that current LLMs struggle significantly with multi-dimensional planning and global optimization, despite agent-based reasoning strategies showing limited improvement.
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers present HOPM, a hierarchical prompt mutation framework that adaptively optimizes language model outputs for high-stakes document generation in marketplace dispute resolution. Testing on 600 real cases, the system achieved an 11 percentage point improvement in win rate and 19.1 percentage point improvement in amount-weighted outcomes compared to static prompting, combining human feedback with automated evaluation.
