ITBench-AA: Frontier Models Score Below 50% on the First Benchmark for Agentic Enterprise IT Tasks — by Artificial Analysis and IBM

Artificial Analysis and IBM released ITBench-AA, the first comprehensive benchmark for evaluating frontier AI models on enterprise IT task automation. The benchmark reveals that leading models score below 50%, exposing significant gaps in agentic AI capabilities for real-world business operations and highlighting the gap between marketing claims and actual performance.

The launch of ITBench-AA represents a critical moment in AI evaluation methodology. Rather than measuring performance on isolated benchmarks like coding or math problems, this benchmark tests how well frontier models handle complex, multi-step enterprise IT workflows—tasks that require planning, tool use, and error recovery. The below-50% performance across leading models signals that agentic AI, despite considerable hype, remains far from production-ready for mission-critical environments.

This benchmark addresses a genuine market need. As enterprises increasingly invest in AI agents for IT operations, security, and infrastructure management, there has been minimal standardized assessment of their actual capabilities. The gap between vendor claims and measured performance creates risk for adopters who may deploy systems with inflated expectations. Prior benchmarks focused on narrow competencies rather than the orchestrated, multi-modal reasoning required for real enterprise tasks.

For the AI industry, ITBench-AA's findings have two competing implications. On one hand, the low scores validate concerns that current models lack reliability for autonomous operation in high-stakes environments, potentially slowing enterprise adoption of agentic systems. On the other hand, the benchmark itself provides a clear improvement roadmap for model developers and establishes measurable goals for the next generation of AI systems. This transparency could accelerate genuine progress rather than incremental marketing cycles.

Market watchers should monitor whether this benchmark becomes industry standard. If enterprises adopt ITBench-AA scores as a decision criteria, companies demonstrating significant improvements will gain competitive advantage. The benchmark may also shift investment focus toward specialized model architectures designed for agentic tasks rather than general-purpose scaling approaches.

• →Frontier AI models score below 50% on ITBench-AA, the first standardized benchmark for enterprise IT task automation
• →The gap between vendor claims and measured agentic performance highlights risks for enterprises deploying AI agents in production
• →Benchmark results provide developers with concrete targets for improving multi-step reasoning and tool orchestration in AI systems
• →ITBench-AA adoption could reshape enterprise AI procurement decisions and shift investment toward specialized agentic architectures
• →Below-50% scores suggest current models require significant refinement before reliable autonomous operation in mission-critical environments