MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoring

Researchers introduce MonitoringBench, a semi-automated red-teaming methodology that reveals significant gaps in AI agent monitoring systems. By decomposing attack generation into strategy, execution, and refinement stages, the team created 2,644 adversarial trajectories showing that frontier monitors claiming 94.9% catch rates actually perform at 60.3% against sophisticated attacks.

MonitoringBench addresses a critical vulnerability in AI safety infrastructure: the systematic underestimation of monitor performance through incomplete adversarial testing. The research reveals that current red-teaming practices suffer from mode collapse, where attack generation becomes repetitive and predictable, ultimately failing to stress-test defensive systems adequately. This matters because autonomous coding agents increasingly handle security-sensitive tasks, and inflated confidence in their monitoring systems creates real deployment risks.

The methodology's innovation lies in decomposing complex attack construction into manageable stages—frontier LLMs excel at either strategy conception or execution individually, but the hybrid approach captures capabilities neither achieves alone. Post-hoc trajectory refinement adds another layer of sophistication. This three-part framework directly addresses why previous benchmarks may have underestimated attack effectiveness, a pattern common across emerging AI safety evaluation.

For the broader AI development ecosystem, this work signals that safety evaluations require adversarial depth matching attacker sophistication. The 34.6 percentage-point performance gap between baseline and refined attacks demonstrates how quickly monitor capabilities can collapse under methodical pressure. Notably, generalization across ten held-out monitors suggests the discovered attack vectors aren't gaming specific implementations but exposing fundamental architectural weaknesses in detection logic.

Moving forward, this benchmark becomes a living evaluation standard as agent capabilities advance. The research highlights that frontier monitors detect suspicious actions but fail at either threat calibration or resist social engineering vectors. Organizations deploying autonomous agents should treat such published benchmarks as minimum baselines rather than confidence signals, expecting real-world attacks to be significantly more effective than published studies indicate.

• →Frontier AI monitors show 34.6% performance drop on refined attacks versus baseline, indicating substantial safety evaluation gaps
• →Semi-automated red-teaming with strategy-execution-refinement decomposition generates more sophisticated and diverse attacks than manual methods
• →Current monitors detect suspicious behavior but struggle with persuasion-based attacks and threat calibration
• →Attack patterns generalize across different monitor architectures, suggesting systemic rather than implementation-specific vulnerabilities
• →MonitoringBench provides reusable methodology for continuously evaluating agent-monitor systems as capabilities evolve