Oversight Has a Capacity: Calibrating Agent Guards to a Subjective, Fatiguing Human

Researchers present an open-source system for overseeing LLM agents taking real-world actions, revealing that human reviewers have only moderate agreement on what constitutes risky behavior and that human fatigue creates an inverted-U safety curve where excessive oversight can paradoxically reduce system safety. The framework reframes agent guardrails as a resource-allocation problem rather than a pure classification challenge.

As large language model agents graduate from text generation to executing irreversible actions like shell commands and file modifications, the AI safety community has converged on a straightforward solution: human-in-the-loop approval gates that pause risky operations for review. This research challenges the assumption that such gates work as intended, uncovering fundamental limitations in how humans evaluate agent behavior.

The core findings strike at the heart of current AI safety practices. When researchers asked multiple human reviewers to label 125 adversarially-weighted agent actions, they achieved only moderate inter-rater agreement (Fleiss' kappa = 0.52), suggesting no objective ground truth for what constitutes "risky" behavior exists. This undermines confidence in building deterministic safety classifiers. More provocatively, the authors model human reviewers as endogenously fatiguing resources—as escalation load increases, reviewer attention degrades. Under this realistic constraint, optimal safety emerges at intermediate escalation rates, not maximum escalation. Counterintuitively, routing every questionable action to humans can degrade overall system safety by exhausting reviewer capacity, allowing genuinely dangerous actions to slip through.

The research has significant implications for deploying autonomous agents in production environments. Organizations building agent systems must now treat human oversight not as an infinite safety valve but as a constrained resource requiring active management. The demonstrated vulnerability to flooding attacks—where adversaries intentionally trigger false escalations to fatigue reviewers—reveals a new attack surface in deployed agent systems. For AI developers and enterprises deploying autonomous systems, this research provides both empirical measurement tools and a framework for calibrating guard policies to realistic human constraints, rather than relying on idealized assumptions about reviewer availability.

• →Human reviewers show only moderate agreement on risky agent actions, indicating no single objective safety standard exists for autonomous systems
• →System safety follows an inverted-U curve with escalation rate—excessive human oversight can reduce safety by exhausting reviewer fatigue
• →Adversaries can exploit fatigued human reviewers through flooding attacks that slip malicious actions past overwhelmed overseers
• →Agent guardrails must be framed as resource-allocation problems with finite human attention rather than perfect classification systems
• →Open-source oversight tools enable measurable calibration of guard policies to realistic human constraints and workload capacities