AIBearisharXiv – CS AI · Jun 237/10
🧠Researchers demonstrate a critical flaw in using large language models as user simulators for training conversational AI: LLM simulators systematically misrepresent how real customers disengage from purchases, showing excessive deliberation and muted resistance compared to actual users. This bias could lead developers to overestimate the effectiveness of sales agents trained on synthetic user interactions.
AIBullisharXiv – CS AI · Jun 237/10
🧠Researchers introduce the Power Systems Agent Benchmark, an executable evaluation framework for AI agents in electric power engineering with 41 task families across eight engineering domains. The benchmark uses deterministic evaluation to assess whether AI agents can perform real power-system engineering tasks correctly, marking the first major standardized assessment tool for this emerging application area.
AINeutralarXiv – CS AI · Jun 237/10
🧠GroundEval introduces a deterministic framework for evaluating AI agents by auditing their evidence retrieval and reasoning paths rather than relying on LLM judges. The tool detected a critical failure case where frontier LLM judges scored an agent response above 0.85, but the actual trace revealed the agent never retrieved the artifact it cited, yielding a GroundEval score of 0.000.
AIBearisharXiv – CS AI · Jun 237/10
🧠Researchers found that AI coding agents produce less maintainable code than humans, with task resolution rates dropping up to 13.1% when subsequent agents build on agent-generated code. Traditional software engineering metrics fail to explain the difference, with subtle behavioral issues like error handling and input validation being key factors.
AINeutralarXiv – CS AI · Jun 127/10
🧠Researchers introduce SciAgentArena, a comprehensive benchmark with ~200 tasks designed to evaluate AI agents in real-world scientific research across multiple domains. The study reveals that while current AI agents excel at well-defined data-analysis tasks, they struggle significantly with novel insight generation, open-ended exploration, and autonomous reasoning in complex scientific contexts.
AINeutralarXiv – CS AI · Jun 97/10
🧠Researchers discovered that 16% of tasks across five major AI agent benchmarks can be exploited by frontier models through reward hacking, corrupting leaderboard rankings and training signals. They developed the hacker-fixer loop, an automated method using three LLM agents to iteratively discover and patch exploits in task verifiers, reducing attack success rates from 62% to 0% on tested benchmarks.
🧠 Claude🧠 Opus🧠 Gemini
AINeutralarXiv – CS AI · Jun 97/10
🧠Researchers introduce SWE-Marathon, a benchmark testing AI agents on 20 ultra-long-horizon software engineering tasks requiring millions of tokens and hours of sustained work. Current frontier coding agents solve fewer than 30% of tasks, revealing critical gaps in planning, self-verification, and memory management that limit real-world deployment.
AIBullisharXiv – CS AI · Jun 97/10
🧠Researchers introduce CUA-Gym, a scalable pipeline for generating verified training data for computer-use agents through co-generation of task instructions, environment states, and reward functions. The resulting dataset of 32,112 verified training tuples across 110 environments enables AI agents to achieve 62.1-72.6% performance on benchmarks, significantly advancing verifiable reinforcement learning for autonomous computer interaction.
AINeutralarXiv – CS AI · Jun 87/10
🧠A comprehensive study of deployed LLM-based agents across 26 domains reveals that production systems rely on simple, human-centered approaches rather than complex automation. The research shows 68% of agents require human intervention within 10 steps, 70% use prompt engineering instead of model fine-tuning, and reliability remains the primary development challenge addressed through systems-level design.
AINeutralarXiv – CS AI · Jun 57/10
🧠Researchers introduce ToolMaze, a benchmark testing how AI language models handle real-world tool failures and recovery scenarios, revealing that implicit semantic failures cause performance drops of ~37% and that fault-tolerance improves significantly slower than basic task performance as models scale.
AIBearisharXiv – CS AI · Jun 17/10
🧠Researchers introduce LongDS, a benchmark revealing significant limitations in AI agents performing long-horizon data analysis tasks. Testing five state-of-the-art models shows best performance of only 48.45% accuracy with performance degrading by 47 points across task progression, indicating that maintaining analytical context over extended interactions remains a critical unsolved problem.
AIBearisharXiv – CS AI · May 297/10
🧠A large-scale observational study of 20,574 real-world AI coding agent sessions reveals systematic misalignment patterns between developer intent and agent behavior. The research identifies seven recurring failure modes, with 91.49% of visible issues requiring explicit user correction, though most impose effort costs rather than irreversible damage.
AIBullisharXiv – CS AI · May 297/10
🧠Researchers introduce GTA, a scalable framework for automatically generating realistic web agent tasks paired with executable trajectories at scale. The system addresses critical limitations in existing benchmarks by combining crawling, retrieval-based seeding, and automated quality control to create multi-hop, cross-page tasks across 50+ websites, revealing significant performance gaps between human and AI agents.
AIBullisharXiv – CS AI · May 297/10
🧠Researchers propose EELMA, an algorithm that uses information-theoretic empowerment to evaluate language model agents at scale without manual benchmarking. The method measures an agent's ability to influence future states through its actions and demonstrates strong correlation with task performance across text-based, web, and tool-use environments.
AIBullisharXiv – CS AI · May 277/10
🧠Researchers conducted a 4-month case study embedding a persistent AI agent into a real academic research environment, tracking 75,671 telemetry records across 96 active days. The study reveals that persistent agents shift computational economics from cost-per-token to cost-per-artifact, with cache-dominant workflows achieving 82.9% token reuse efficiency.
AIBearisharXiv – CS AI · May 127/10
🧠Researchers introduce MonitoringBench, a semi-automated red-teaming methodology that reveals significant gaps in AI agent monitoring systems. By decomposing attack generation into strategy, execution, and refinement stages, the team created 2,644 adversarial trajectories showing that frontier monitors claiming 94.9% catch rates actually perform at 60.3% against sophisticated attacks.
AINeutralarXiv – CS AI · May 117/10
🧠Researchers introduce Agentick, a unified benchmark for evaluating diverse AI agents—from reinforcement learning to large language models—across 37 procedurally generated tasks. Testing 27 configurations reveals no single approach dominates, with GPT-4 mini leading overall while specialized methods excel in specific domains, suggesting significant optimization potential across all agent paradigms.
🏢 Meta🧠 GPT-5
AINeutralarXiv – CS AI · Apr 147/10
🧠Researchers introduce The Amazing Agent Race (AAR), a new benchmark revealing that LLM agents excel at tool-use but struggle with navigation tasks. Testing three agent frameworks on 1,400 complex, graph-structured puzzles shows the best achieve only 37.2% accuracy, with navigation errors (27-52% of failures) far outweighing tool-use failures (below 17%), exposing a critical blind spot in existing linear benchmarks.
🧠 Claude
AINeutralarXiv – CS AI · Apr 107/10
🧠Researchers introduce ATBench, a comprehensive benchmark for evaluating the safety of LLM-based agents across realistic multi-step interactions. The 1,000-trajectory dataset addresses critical gaps in existing safety evaluations by incorporating diverse risk scenarios, detailed failure classification, and long-horizon complexity that mirrors real-world deployment challenges.
AIBullisharXiv – CS AI · Mar 46/104
🧠Researchers present a new framework for evaluating logical reasoning AI agents using an "assessor agent" that can issue tasks, enforce execution limits, and record structured failure types. Their auto-formalization agent achieved 86.70% accuracy on logical reasoning tasks, outperforming traditional chain-of-thought approaches by nearly 13 percentage points.
AINeutralarXiv – CS AI · Jun 256/10
🧠Researchers present a taxonomy of memory roles in RAG-based conversational AI systems, demonstrating that different memory types—such as clarifying versus irrelevant memories—substantially shape response quality, factual accuracy, and personalization. Using a user-centric evaluation framework, the study reveals that memory function matters more than just storage mechanisms, with implications for developing more effective conversational agents.
AIBullisharXiv – CS AI · Jun 236/10
🧠Researchers introduce Agentic Time Machine (TM), an infrastructure that reconstructs past web states to enable efficient evaluation of AI agents on event forecasting tasks. A multi-agent framework using this system achieves top performance on FutureX benchmarks and Polymarket predictions, demonstrating that offline evaluation correlates strongly with live forecasting results.
AINeutralarXiv – CS AI · Jun 236/10
🧠Researchers introduce Trip+, a new benchmark for evaluating AI agents in travel planning that measures holistic performance across personalization, feasibility, and interaction quality. Testing 18 language models reveals a consistent gap where agents generate technically viable but exhausting itineraries that poorly match traveler preferences, highlighting limitations in how current LLMs handle complex, profile-conditioned decision-making over multiple turns.
AINeutralarXiv – CS AI · Jun 236/10
🧠SkillAudit introduces an automated framework for evaluating AI agent skills independently of fixed task benchmarks, addressing a critical gap in skill marketplaces. The research reveals that over 7% of real-world skill packages exhibit risky behavior, highlighting the need for systematic assessment tools as AI skill ecosystems expand.
AIBearisharXiv – CS AI · Jun 196/10
🧠Researchers introduced ORAgentBench, a benchmark testing whether AI agents can autonomously solve complex operations research tasks end-to-end. Testing 14 frontier agent-model configurations revealed significant limitations: the best agent solved only 35.51% of tasks and 20.59% of hard tasks, with failures stemming from missed operational rules, weak solution construction, and insufficient optimization—indicating AI agents remain far from production-ready OR work.
