AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts

Researchers introduce AgencyBench, a comprehensive benchmark for evaluating autonomous AI agents across 32 real-world scenarios requiring up to 1 million tokens and 90 tool calls. The evaluation reveals closed-source models like Claude significantly outperform open-source alternatives (48.4% vs 32.1%), with notable performance variations based on execution frameworks and model optimization.

AgencyBench addresses a critical gap in AI agent evaluation by moving beyond single-capability testing toward complex, long-horizon real-world tasks. Traditional benchmarks fail to capture how autonomous agents operate in production environments where agents must orchestrate multiple tool calls, handle iterative feedback, and sustain execution over extended periods. This research matters because it reveals meaningful performance disparities between proprietary and open-source models—gaps that may reflect architectural advantages, training data quality, or integration optimization rather than raw capability differences.

The benchmark's use of automated evaluation through user simulation agents and Docker-based assessment sidesteps the scalability bottleneck of human-in-the-loop feedback, enabling systematic testing at scale. The finding that closed-source models achieve 48.4% task completion versus 32.1% for open-source models has significant implications for enterprise adoption decisions. However, the discovery that open-source models show distinct performance peaks in specific frameworks suggests optimization potential rather than fundamental capability ceilings.

For the AI developer ecosystem, this research underscores an emerging trend: agent performance depends not solely on model quality but on co-optimization between model architecture and execution frameworks. Claude's superior performance within its native SDK ecosystem demonstrates this principle. This finding likely accelerates investment in agent-specific frameworks and suggests that open-source models could narrow performance gaps through better integration engineering.

Looking forward, AgencyBench establishes a standardized evaluation methodology that the industry can adopt, potentially accelerating agent development and making performance comparisons more meaningful. The open-source release democratizes testing while inviting community efforts to improve open-source agent performance.

• →AgencyBench introduces the first comprehensive benchmark evaluating autonomous agents on 138 real-world tasks requiring up to 1 million tokens and hours of execution.
• →Closed-source models significantly outperform open-source models (48.4% vs 32.1%), though performance varies substantially based on framework optimization.
• →Agent performance depends on co-optimization between model architecture and execution frameworks rather than model quality alone.
• →Automated evaluation using user simulation agents and Docker sandboxes enables scalable testing without human-in-the-loop bottlenecks.
• →Open-source models show distinct performance peaks in specific frameworks, suggesting optimization pathways to narrow the proprietary model gap.