GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows

Researchers introduce GTA-2, a hierarchical benchmark that evaluates AI agents on both atomic tool-use tasks and complex, open-ended workflows using real user queries and deployed tools. The study reveals a significant capability cliff where frontier AI models achieve below 50% success on atomic tasks and only 14.39% on realistic workflows, highlighting that execution framework design matters as much as underlying model capacity.

GTA-2 addresses a critical gap between how AI agents are currently evaluated and how they must perform in production environments. Previous benchmarks relied on synthetic queries and dummy tools, creating an unrealistic assessment of agent capabilities. This new framework uses authentic user queries, real-world tools, and multimodal contexts, providing researchers and developers with meaningful performance metrics that reflect actual deployment challenges.

The research reveals a stark capability cliff that challenges assumptions about frontier model performance. While leading models already struggle on simple, closed-ended tool-use tasks, their performance collapses on realistic workflows requiring multi-step reasoning and coordination. The 14.39% success rate on open-ended tasks demonstrates that moving from constrained to real-world scenarios exposes fundamental limitations in current AI agent architectures.

The finding that execution frameworks like Manus and OpenClaw substantially improve workflow completion rates has significant implications for AI development priorities. Rather than focusing exclusively on scaling model parameters, the results suggest that engineering better execution harnesses—the orchestration layer managing tool calls and state management—yields measurable improvements. This shifts investment focus toward middleware and framework design for practical AI applications.

For developers building personal and professional assistants, these benchmarks establish realistic expectations and highlight where bottlenecks exist. The recursive checkpoint-based evaluation mechanism provides a replicable methodology for assessing agent progress, enabling teams to measure incremental improvements in complex task completion. Future development should balance model capability improvements with architectural innovations in agent execution frameworks.

• →Frontier AI models achieve only 14.39% success on open-ended workflow tasks despite near-50% performance on atomic tool use, revealing a critical capability gap.
• →Execution framework design (harnesses like Manus and OpenClaw) substantially improves workflow completion, suggesting equal importance to model capacity for production agents.
• →Real-world benchmarking with authentic queries and deployed tools exposes significant limitations that synthetic evaluation datasets mask.
• →Checkpoint-guided feedback mechanisms enable decomposition of complex objectives into verifiable sub-goals, improving agent performance and evaluation reliability.
• →The capability cliff between atomic and workflow tasks indicates current AI agents lack sufficient multi-step reasoning and system-level coordination for realistic productivity applications.