When Evidence is Sparse: Weakly Supervised Early Failure Alerting in Dialogs and LLM-Agent Trajectories

Researchers present a weakly supervised approach for detecting dialog and agent failures early in their execution, introducing an attention-based predictor that identifies sparse failure evidence and pairs it with a preference-conditioned stopping policy. The method achieves 3-42% improvement over existing approaches while reducing training costs by 1-3 orders of magnitude across five benchmarks.

Early failure detection in dialog systems and AI agents represents a critical challenge as these systems become increasingly deployed in customer-facing applications. This research addresses a fundamental mismatch between how supervision signals are typically provided—as binary success/failure labels for complete trajectories—and the practical need to alert systems during ongoing interactions. The key insight that failure evidence is sparse, occupying only 4.7-11.3% of conversation turns and often appearing late in interactions, challenges conventional approaches that naively propagate terminal labels to all conversation prefixes.

The proposed solution leverages an attention mechanism to learn which turns actually indicate failure risk rather than treating all turns equally. This sparse learning approach mirrors how humans might identify problems during conversations—by recognizing specific signals rather than viewing every moment as equally diagnostic. The accompanying α-STOP policy enables flexible operating point selection at inference time, eliminating the need to retrain separate detection models for different accuracy-earliness trade-offs.

For practitioners building dialog systems and autonomous agents, this research provides practical technical improvements. The dramatic reduction in training costs while maintaining superior performance makes the approach economically attractive for scaling. The Pareto-frontier improvements demonstrate measurable gains across diverse interaction types—customer support, task-oriented dialog, persuasion, tool use, and planning—suggesting broad applicability.

Future development likely centers on extending these methods to longer, more complex agent trajectories and integrating failure detection with active remediation strategies rather than static alerting.

• →Failure evidence in multi-turn interactions is sparse (4.7-11.3% of turns) and delayed, invalidating common prefix-label supervision approaches
• →Attention-based predictors improve Pareto-frontier quality by 1-10% by learning genuine failure signals rather than treating all turns identically
• →The full system achieves 3-42% frontier improvements over state-of-the-art while reducing per-operating-point training costs by 1-3 orders of magnitude
• →A single preference-conditioned stopping policy enables flexible accuracy-earliness trade-offs at inference without retraining separate models
• →The approach generalizes effectively across five diverse benchmarks spanning customer support, task-oriented dialog, persuasion, tool use, and planning