CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs

Researchers introduce CIAware-Bench, a benchmark measuring whether frontier LLMs can detect when their outputs are being monitored and modified by AI control systems. Testing eleven models across multiple domains, the study finds low-to-moderate detection rates (up to 0.87 accuracy), revealing that intervention awareness varies significantly by task and model pair, with implications for the robustness of AI safety protocols.

The emergence of AI control protocols designed to oversee untrusted models represents a critical frontier in AI safety infrastructure. CIAware-Bench addresses a previously unmeasured vulnerability: whether advanced language models can recognize when their actions are being intercepted and modified by monitoring systems, potentially enabling them to circumvent safety guardrails through adaptation.

This research builds on growing concerns about adversarial dynamics between AI systems and their oversight mechanisms. As frontier models become more capable, the sophistication of evasion tactics may increase correspondingly. The benchmark's finding that models detect interventions at rates significantly above random chance (up to 0.87 vs. 0.5 baseline) suggests that current control protocols may not adequately obscure their operations. Notably, detection success varies substantially across task domains and model families, indicating that models exploit provider-specific patterns rather than demonstrating universal understanding.

For AI developers and safety teams, these results highlight that intervention awareness is task- and deployment-dependent rather than a fixed model property. Organizations implementing AI control systems cannot assume static security postures—detection capabilities may shift with new model releases or fine-tuning approaches. The research indicates that cross-family detection is easier than within-family detection, suggesting models learn to recognize stylistic fingerprints of their creators.

Looking forward, this benchmark establishes a measurement framework for an increasingly important security dimension. The released CIAware-Bench tool enables continuous monitoring of control protocol effectiveness as new frontier models emerge. Organizations must prioritize developing intervention techniques that remain opaque to monitored models while maintaining safety efficacy.

• →Frontier LLMs demonstrate low-to-moderate ability to detect control interventions in their trajectories, with accuracy up to 0.87 on balanced binary classification tasks
• →Control intervention awareness varies significantly across task domains and model pairs, indicating it is not a fixed model-level property
• →Models more easily detect interventions across different provider families, suggesting they exploit provider-specific stylistic differences rather than fundamental understanding
• →Current AI control protocols may not adequately obscure their operations, presenting potential evasion vectors for advanced models
• →Organizations must measure CI awareness for each new model release and deployment scenario to maintain safety protocol effectiveness