ADK Arena: Evaluating Agent Development Kits via LLM-as-a-Developer

Researchers introduce ADK Arena, an automated evaluation framework that uses LLMs as proxy developers to benchmark 51 Python Agent Development Kits across multiple benchmarks. The study reveals significant performance variation across frameworks, with generation costs varying 5.6x and no single dominant framework, while documentation and source code prove largely substitutable in agent development.

The fragmentation of the Agent Development Kit landscape has created a critical gap: developers lack empirical data on framework performance relative to implementation complexity. ADK Arena addresses this through a novel methodology where LLM agents serve as consistent developers, isolating framework effects from human skill variation. By measuring generation cost as a proxy for API usability, the research provides quantitative insights into developer experience across 51 frameworks—a comprehensive evaluation that would be prohibitively expensive with human developers.

The findings challenge several assumptions in the AI agent development community. The 5.6x cost variance across frameworks indicates significant API design differences, yet cost alone doesn't predict task resolution success. More intriguingly, the 28-40% framework usage band across information-source ablations suggests developers don't fundamentally need perfect documentation—parametric knowledge in LLMs compensates for missing reference materials. This democratizes framework accessibility and reduces documentation quality as a competitive differentiator.

The results have immediate implications for framework maintainers and enterprise adoption decisions. While leading frameworks achieve 80% task resolution on some benchmarks, the median framework reaches only 32%, revealing substantial quality disparities. This creates opportunity for emerging frameworks to differentiate through superior API design and developer experience. For organizations building autonomous agent systems, the study quantifies trade-offs between framework sophistication and implementation difficulty, enabling more informed technology selection based on specific use cases rather than hype cycles.

• →Generation costs vary 5.6x across 51 Python ADK frameworks ($0.6-$3.4 per agent), revealing substantial API complexity differences
• →No single framework dominates across benchmarks; top performers achieve 80% task resolution while median frameworks reach only 32%
• →LLM parametric knowledge substitutes effectively for documentation—framework usage stays within 28-40% band regardless of information source availability
• →Best single-benchmark ADK agents outperform general-purpose frontier coding agents at a fraction of the cost on specific tasks
• →API design complexity doesn't guarantee task performance, requiring comprehensive empirical evaluation rather than feature-count comparisons