Computer Use at the Edge of the Statistical Precipice

Researchers expose critical flaws in Computer Use Agent (CUA) benchmarking, demonstrating that simple replay scripts outperform advanced AI models on current static benchmarks. The study introduces PRISM design principles and DigiWorld, a rigorous evaluation framework with 3.2 million verified configurations, establishing new standards for meaningful CUA assessment.

The research identifies a fundamental crisis in how Computer Use Agents are evaluated. A trivial 1MB replay script that memorizes action sequences without processing visual feedback achieves higher benchmark scores than frontier AI models, exposing the methodological bankruptcy of current evaluation approaches. This discovery reveals that existing benchmarks fail to test genuine agent reasoning and adaptability.

The field has grown rapidly without establishing principled evaluation standards. Most current benchmarks use static, unsandboxed environments with unreliable verification, creating conditions where memorization outperforms intelligence. The misapplication of pass@k metrics designed for sampling tasks to stateful UI interactions compounds this problem, leading researchers to draw false conclusions about agent capabilities.

DigiWorld addresses these failures through the PRISM framework: privileged verification ensures ground truth, realistic environments prevent overfitting to artificial scenarios, integrity-checked configurations maintain consistency, sandboxed execution enables safe testing, and multifactorial variability prevents memorization. The benchmark's 3.2 million unique verified configurations create sufficient diversity to distinguish genuine agent intelligence from replay attacks.

This research carries significant implications for AI development teams and enterprises evaluating CUAs for deployment. Companies relying on current benchmark rankings may be selecting agents that merely memorize common task patterns rather than demonstrating genuine reasoning. As autonomous systems increasingly handle consequential tasks, rigorous evaluation becomes non-negotiable. The work establishes that methodological rigor is not an academic luxury but a prerequisite for developing trustworthy autonomous agents. Future CUA development must adopt these standards to generate reliable performance claims.

• →Simple replay scripts outperform frontier models on current CUA benchmarks, exposing systematic evaluation failures
• →Static, unsandboxed environments and misused metrics enable memorization to masquerade as agent intelligence
• →PRISM principles and DigiWorld benchmark provide 3.2 million verified configurations for genuine capability assessment
• →Current CUA evaluation methodology fails to distinguish intelligent reasoning from pattern memorization
• →Rigorous benchmarking standards are prerequisites for trustworthy autonomous agent deployment in real-world applications