#agent-benchmarking News & Analysis

3 articles tagged with #agent-benchmarking. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

3 articles

AINeutralarXiv – CS AI · May 287/10

🧠

A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks

Researchers introduce TASTE, an automated method for generating challenging AI agent benchmarks by reversing traditional task construction—starting from tool sequences rather than natural language descriptions. The resulting τc-Bench significantly increases difficulty and tool-use diversity, revealing that high performance on existing saturated benchmarks like τ2-Bench doesn't guarantee robust agent capabilities.

🧠 Gemini

AINeutralarXiv – CS AI · Jun 106/10

🧠

CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks

Researchers introduce CollabSkill, a framework for evaluating how AI agents perform when collaborating with real human workers on occupational tasks. Using data from 93 workers across 386 sessions, the study reveals that Claude Code outperforms Codex in practical collaboration scenarios—diverging from autonomous benchmark rankings—and identifies hands-on experience as the primary driver of effective human-AI teamwork.

🧠 Claude

AINeutralarXiv – CS AI · Jun 96/10

🧠

Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agents

Researchers propose Online Agent-as-a-Judge, a new evaluation framework that uses an in-world evaluator agent to actively test LLM-powered interactive agents across specific social scenarios. Unlike passive evaluation methods, this approach generates targeted situations to reveal behaviors that might otherwise remain unobserved, improving assessment reliability in complex multi-agent environments.