Do More Agents Help? Controlled and Protocol-Aligned Evaluation of LLM Agent Workflows

Researchers introduce BenchAgent, an evaluation framework comparing single-agent and multi-agent LLM workflows under standardized conditions across ten benchmarks. Results show that adding more agents does not consistently improve performance, with only one of six tested multi-agent systems exceeding single-agent baselines, while most incur higher computational costs for lower accuracy.

The research challenges a prevailing assumption in AI development: that orchestrating multiple specialized agents inherently produces better outcomes than single-agent systems. By establishing BenchAgent—a normalized evaluation protocol ensuring all workflows share identical tool access, logging, and accounting mechanisms—the researchers eliminate confounding variables that typically plague multi-agent comparisons. This methodological rigor reveals that complexity does not guarantee performance gains.

The findings reflect ongoing debates about agentic AI architecture choices. As LLM-powered agents gain adoption for reasoning and tool-use tasks, organizations face pressure to adopt multi-agent systems without empirical evidence of superiority. The study's controlled conditions expose that five of six tested multi-agent systems underperformed their single-agent counterparts by 2.56-11.29 percentage points while consuming more computational resources, creating unfavorable accuracy-to-cost trade-offs. Only EvoAgent's performance remained competitive with single-agent solutions.

For AI developers and enterprise adopters, this work suggests that multi-agent complexity warrants stronger justification than intuitive appeal. The runtime-generated Claude workflow's strong GAIA performance (66.72% overall) indicates that dynamic, protocol-aligned agent composition may outperform fixed multi-agent designs, pointing toward adaptive architectures rather than static team structures. This challenges infrastructure vendors and framework creators to prioritize flexible agent routing over rigid orchestration layers.

The implications extend to AI budgeting decisions: organizations pursuing multi-agent strategies should demand comparable benchmarking rather than assuming architectural novelty translates to capability gains. Future research should explore why certain dynamic workflows like Claude's succeed where fixed multi-agent systems plateau.

• →Adding more agents to LLM workflows does not guarantee performance improvements and often reduces efficiency on standard benchmarks
• →Only one of six tested multi-agent systems exceeded single-agent performance under controlled evaluation conditions
• →Five multi-agent approaches underperformed by 2.56-11.29 percentage points while consuming more computational resources
• →Dynamic, runtime-generated agent architectures show stronger results than fixed multi-agent designs on complex reasoning tasks
• →Standardized evaluation protocols are essential for accurate comparison of agentic AI systems