AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents

Researchers introduced AD-Bench, a real-world benchmark for evaluating LLM agents in advertising analytics tasks using actual production platform data. The framework addresses the gap between idealized benchmarks and practical agent performance, revealing that state-of-the-art models like Claude-Opus-4.7 struggle significantly with complex, multi-step advertising analytics despite achieving 76.9% accuracy on simpler tasks.

AD-Bench represents a meaningful shift in how AI agents are evaluated beyond theoretical settings. The benchmark tackles a genuine problem in LLM evaluation: existing benchmarks rely on static, idealized scenarios that don't capture the complexity of specialized professional domains. Real advertising analytics requires continuous tool interaction, multi-round reasoning, and adaptation to evolving data—dynamics that most current evaluation frameworks ignore entirely.

The research emerges from growing recognition that LLM agents, despite impressive capabilities on general reasoning tasks, struggle when deployed in production environments with domain-specific requirements. Advertising analytics demands understanding platform-specific metrics, navigating complex query systems, and maintaining consistency across tool calls. Traditional benchmarks fail to measure these capabilities because they use frozen, simplified scenarios. AD-Bench's innovation—a dynamic ground-truth pipeline that regenerates correct answers based on current platform state—directly addresses answer obsolescence, a critical failure mode in production settings.

The performance data reveals important limitations in current agent architectures. Claude-Opus-4.7's dramatic drop from 80.4% (Pass@3 on overall tasks) to 65.1% (Pass@3 on hardest tasks) demonstrates that scaling alone hasn't solved complex reasoning in specialized domains. The 82.7% trajectory coverage metric introduces trajectory-aware evaluation, moving beyond simple end-to-end correctness to measure reasoning process quality—a crucial distinction for production systems.

This benchmark will likely influence how AI development teams assess agent readiness for enterprise deployment. The framework suggests future evaluation methodologies should prioritize dynamic environments, domain-specific tools, and process-level analysis rather than static benchmarks. Organizations building LLM agents for professional analytics tools now have concrete evidence that substantial gaps remain, even with leading models.

• →AD-Bench benchmarks LLM agents on real-world advertising analytics tasks, revealing state-of-the-art models achieve only 61-65% accuracy on complex problems.
• →Dynamic ground-truth pipeline regenerates correct answers based on current platform state, solving the answer-obsolescence problem in evolving production environments.
• →Trajectory-aware evaluation measures both correctness and reasoning process quality, a more practical metric for production agent deployment.
• →Claude-Opus-4.7 shows 15-19 percentage point performance degradation on hardest tasks, indicating fundamental limitations in current agent architectures.
• →The benchmark emphasizes that specialized domains with multi-tool requirements remain significantly challenging despite advances in general-purpose LLM reasoning.