Act or Escalate? Evaluating Escalation Behavior in Automation with Language Models

Researchers analyzed how large language models decide whether to act on predictions or escalate to humans, finding that models use inconsistent and miscalibrated thresholds across five real-world domains. Supervised fine-tuning on chain-of-thought reasoning proved most effective at establishing robust escalation policies that generalize across contexts, suggesting escalation behavior requires explicit characterization before AI system deployment.

This research addresses a critical gap in AI automation systems: determining when AI should defer to human judgment. The study frames escalation as a decision-making problem where models must weigh the costs of acting incorrectly against the costs of escalating, using domains like loan approval, content moderation, and autonomous driving where such decisions carry real consequences. The finding that different model architectures and scales exhibit fundamentally different escalation thresholds challenges assumptions about model consistency and reveals that larger or more sophisticated models don't automatically make better escalation decisions. The miscalibration of self-estimates—where models' confidence in their own accuracy doesn't match reality—presents a significant safety concern for deployed systems. Supervised fine-tuning approaches that explicitly train models to reason about uncertainty and decision costs outperformed simpler prompting interventions, suggesting that robust AI behavior requires targeted training rather than architectural changes. This work has immediate implications for AI safety and deployment practices. Organizations implementing AI automation systems cannot rely on default model behavior for critical decisions; they must actively characterize and train models to handle escalation appropriately for their specific cost structures. The generalization of SFT-trained policies across datasets and domains suggests a scalable path forward. As AI systems increasingly handle high-stakes decisions in finance, healthcare, and safety-critical applications, establishing reliable escalation mechanisms becomes foundational to responsible deployment rather than an afterthought.

• →Language models use inconsistent and unpredictable thresholds for deciding when to escalate decisions to humans, varying by model without correlation to architecture or scale.
• →Self-confidence estimates in LLMs are miscalibrated in model-specific ways, making them unreliable indicators for escalation decisions without additional training.
• →Supervised fine-tuning on chain-of-thought reasoning yields the most robust escalation policies that generalize across datasets, cost ratios, and different domains.
• →Escalation behavior must be explicitly characterized and trained before deploying AI systems in high-stakes domains like loan approval and autonomous driving.
• →Prompting interventions alone provide limited effectiveness for establishing reliable escalation behavior compared to explicit model training on decision costs.