Calibration Is Not Control: Why LLM-Agent Oversight Needs Intervention

Researchers argue that current LLM agent oversight systems rely on flawed scalar risk prediction rather than intervention-aware decision-making. Their framework measures intervention advantage—the actual utility gain from intervening—and demonstrates that action-conditioned control significantly outperforms traditional calibrated risk scoring across multiple benchmarks.

The paper identifies a fundamental mismatch in how AI oversight systems are designed. Current approaches treat agent monitoring as a calibration problem: predict failure risk, set a threshold, intervene when crossed. This assumes that identical risk scores warrant identical responses, but the authors demonstrate this assumption breaks down in practice. Two trajectories with identical failure probabilities may differ critically in recoverability—one may be salvageable through intervention while the other has already become irrecoverable. The prefix branching methodology introduced here executes candidate interventions from identical states to measure actual intervention value rather than theoretical risk.

This research addresses a growing concern in AI deployment: as language models become autonomous agents handling real-world tasks, oversight mechanisms must shift from passive risk monitoring to active value assessment. Traditional calibration improvements—making risk scores more accurate—do nothing to fix the fundamental targeting problem. The experiments demonstrate this starkly: recalibrating scalar scores improved prediction metrics while leaving control regret unchanged, proving calibration alone cannot bridge the gap.

For AI developers and deployment teams, this implies substantial gains are available through smarter intervention strategies. The ALFWorld benchmark shows control regret dropping from 0.506 to 0.110 with action-conditioned approaches, representing more than 78% improvement. These gains depend on intervention strength and available information, suggesting that different deployment contexts require tailored control strategies rather than one-size-fits-all risk thresholding. The research direction toward value estimation rather than risk calibration represents a meaningful evolution in how autonomous systems should be monitored and controlled.

• →Scalar risk prediction misses the actual decision problem in LLM agent oversight—intervention utility matters more than failure likelihood.
• →Identical risk scores can require opposite actions depending on trajectory recoverability, creating systematic control failures in current systems.
• →Action-conditioned control reduces control regret by 78% on benchmarks compared to calibrated risk scoring alone.
• →Calibration improvements do not fix target error, meaning better risk predictions alone cannot repair oversight effectiveness.
• →The shift from risk scoring to value estimation represents a fundamental rethinking of autonomous agent oversight architecture.