Early Diagnosis of Wasted Computation in Multi-Agent LLM Systems via Failure-Aware Observability

Researchers introduce a failure-aware observability framework to diagnose wasted computation in multi-agent LLM systems, identifying six failure modes through online trace signals. Testing on 165 GAIA validation traces reveals 41% failure rates across difficulty levels and token consumption ranging from 8,152 to 16,389 tokens, positioning observability as a diagnostic layer between execution logs and accuracy.

This research addresses a critical inefficiency in multi-agent LLM systems: the inability to detect when a computation trajectory has become unrecoverable until final evaluation occurs. Rather than waiting for complete task failure, the proposed framework enables early diagnosis through six observable failure modes—tool reliability issues, execution recovery failures, orchestration loops, evidence scarcity, information staleness, and resource exhaustion. This represents a shift toward proactive rather than reactive optimization in agentic AI systems.

The empirical findings underscore substantial computational waste in current architectures. Across 165 test traces, failure rates of 22-46% per difficulty tier demonstrate systemic brittleness. More revealing is the divergence between token consumption and evidence quality: while higher difficulty tasks require nearly double the tokens, evidence availability and ground-truth support don't scale proportionally. This suggests current systems amplify computation without proportional capability gains.

For AI infrastructure developers and enterprise deployments, this framework directly impacts operational costs. Multi-agent systems running on paid API endpoints accumulate waste through failed retry loops and dead-end tool calls. Early failure detection enables circuit-breaking mechanisms—automatically halting unproductive paths rather than exhausting allocated compute budgets. The finding that 'cheap online signals and deeper semantic metrics capture complementary layers' suggests hybrid monitoring approaches combining lightweight statistical checks with periodic LLM-based audits.

Future development hinges on whether frameworks like this transition from academic observability into production monitoring. If adopted in commercial AI platforms, early failure detection could reduce per-task costs by 20-40% while improving user experience through faster feedback loops.

• →Multi-agent LLM systems fail 22-46% of the time despite consuming 8,152-16,389 tokens, indicating significant computational waste before final-answer evaluation.
• →Six recurring failure modes—tool reliability, orchestration loops, evidence scarcity, and others—can be detected via online trace signals rather than waiting for task completion.
• →Token consumption rises substantially with task difficulty, but evidence quality and support diverge, suggesting systems amplify computation without proportional capability gains.
• →Hybrid monitoring combining lightweight online signals and semantic LLM-based audits captures complementary failure-detection layers.
• →Early failure diagnosis enables circuit-breaking mechanisms that could reduce per-task costs while improving response time for end users.