Domain-Conditioned Safety in Frontier Computer-Using Agents: A 793-Episode Browser Benchmark, a Coding-Domain Cross-Reference, and a Reproducibility Audit of Recent Red-Teaming

Researchers challenge the credibility of recent computer-using agent (CUA) red-teaming studies by reproducing published prompt-injection attacks against frontier models Claude Sonnet 4.6 and GPT-5.4, finding 0% success rates compared to reported 42-98% attack success rates in prior work. The analysis reveals that published high attack success rates depend on reinforcement-learning optimized injection text rather than fundamental attack categories, and that safety hardening is domain-specific to browser interfaces, not generalizable across CUA modalities.

This research exposes a significant reproducibility crisis in AI safety literature surrounding computer-using agents. The authors systematically tested hand-crafted attack templates from prominent red-teaming papers against current frontier models and found near-zero success, contradicting headline numbers claiming 42-98% attack success rates. The gap stems from a critical methodological issue: published papers report results from RL-optimized injection strings rather than replicable attack categories, making their findings unreproducible by external researchers.

The work reflects growing pains in the emerging field of CUA security. As AI systems gain capability to interact with web interfaces and code execution environments, understanding actual vulnerability surfaces becomes crucial for deployment decisions. However, the current literature conflates two distinct concepts—the sophistication of optimization techniques versus the fundamental robustness of model safeguards—obscuring what security improvements genuinely matter.

The domain-conditioned finding carries immediate practical implications. Frontier models demonstrate strong resistance to browser-based prompt injection but remain vulnerable to skill-injection attacks in coding contexts, revealing that safety hardening follows capability boundaries rather than representing holistic robustness. This fragmentation suggests companies may be over-invested in securing heavily-publicized attack vectors while neglecting equally viable exploitation routes in less-discussed modalities.

Moving forward, the field requires standardized, reproducible benchmarks with released attack artifacts and clear separation between attack methodology and optimization technique. Researchers must avoid extrapolating domain-specific safety claims without empirical validation across multiple CUA surfaces. This work's emphasis on reproducibility and transparent methodology could reshape how the community evaluates AI safety claims.

• →Hand-crafted attack templates from published red-teaming papers achieve 0% success against Claude Sonnet 4.6 and GPT-5.4, contradicting prior claims of 42-98% attack success rates.
• →High reported attack success rates depend primarily on reinforcement-learning optimized injection text rather than fundamental attack categories, making published results difficult to reproduce.
• →Frontier model safety hardening is domain-conditioned, providing strong resistance to browser-based prompt injection but remaining vulnerable to skill-injection in coding environments.
• →The literature's methodology conflates optimization sophistication with attack efficacy, obscuring true security boundaries and misleading deployment decisions.
• →Standardized, reproducible benchmarks with released attack artifacts are essential to establish credible safety evaluation standards for computer-using agents.