NetArena: Dynamic Benchmarks for AI Agents in Network Automation

NetArena introduces a dynamic benchmarking framework for evaluating AI agents in network automation tasks, addressing limitations of static benchmarks through runtime query generation and network emulator integration. The framework reveals that AI agents achieve only 13-38% performance on realistic network queries, significantly improving statistical reliability by reducing confidence-interval overlap from 85% to 0%.

• →NetArena is a dynamic benchmark generation framework that addresses contamination risks and statistical variance issues in AI agent evaluation for network operations.
• →The framework enables unlimited query generation at runtime and integrates with network emulators to measure correctness, safety, and latency.
• →AI agents demonstrated poor performance on realistic network tasks, achieving only 13-38% average performance with some queries as low as 3%.
• →NetArena reduced confidence-interval overlap from 85% to 0%, significantly improving statistical reliability across AI agent benchmarks.
• →The framework supports advanced AI training methods including supervised fine-tuning and reinforcement learning for network system tasks.