Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents

A study of a deployed food-and-beverage ordering chatbot reveals that LLM-based quality judges catch fewer than 25% of genuine defects, missing systematic failures in state-tracking and multi-turn consistency while excelling only at single-turn issues. The research demonstrates that automated evaluation metrics are fundamentally insufficient for production multi-agent systems and should not replace human review.

The research exposes a critical vulnerability in the AI evaluation infrastructure underpinning production conversational systems. While the industry routinely reports LLM-judge reliability through agreement metrics with human raters, this study demonstrates the metric masks catastrophic blind spots: a judge flagged zero defects in a batch containing 23 confirmed problems across 7 distinct failure patterns. The root cause is architectural rather than perceptual—the scoring rubric omits behavioral dimensions entirely, routing defects about state-tracking and guardrails into catch-all categories like "brand voice" rather than flagging them as operational failures.

This finding challenges a widespread assumption in AI development: that sufficiently sophisticated language models can self-evaluate their own outputs. The taxonomy of failures reveals a stark pattern—judges detect local issues (fabricated statistics, wrong language) but systematically miss cross-turn dependencies (cart hallucinations, confirm-gate lockouts, stale references). These failures cluster in precisely the areas most critical to user trust: whether the system maintains coherent state across conversation turns.

For the broader AI industry, the implications are sobering. Production systems relying on automated quality gates without human validation operate with fundamentally invisible failure modes. The study's finding that apparent defect rates of zero cannot be corrected using standard statistical estimators suggests that many deployed systems reporting excellent automated quality scores may harbor significant latent problems. Organizations building multi-turn agents must recognize that automation introduces systematic blind spots that scale with system complexity rather than being solved by it.

• →LLM judges catch only 22% of confirmed defects in production ordering agents, missing systematic cross-turn state failures entirely.
• →Judge failures cluster in behavioral dimensions like state-tracking and guardrails, which are absent from current evaluation rubrics.
• →Automated quality gates routed 113 of 114 detected state defects into wrong categories, creating zero operational flags despite confirmed problems.
• →When automated judges report zero defects, statistical correction methods fail—no mathematical adjustment can recover true defect rates from null signals.
• →Human review remains mandatory for production multi-turn agents; automated judging functions as a floor metric, not a substitute.