MedCUA-Bench: A Screenshot-Only Benchmark for Clinical Computer-Use Agents

Researchers introduced MedCUA-Bench, a new benchmark for evaluating AI agents performing clinical computer tasks across 18 medical scenarios. The benchmark reveals significant performance gaps, with top closed-source models achieving only 54.2% success and open-source agents averaging just 2.5%, highlighting the unpreparedness of current AI systems for reliable medical software automation.

MedCUA-Bench addresses a critical validation gap in AI development by establishing the first comprehensive benchmark specifically designed for clinical computer-use agents. While general-purpose AI benchmarks have proliferated, the medical domain requires unique evaluation criteria that traditional task-completion metrics fail to capture. This benchmark reconstructs authentic clinical interfaces from real product manuals and open-source systems, enabling testing without licensing constraints or privacy violations—a pragmatic approach to medical AI validation.

The performance data reveals a stark reality: current AI agents are fundamentally unprepared for clinical environments. Closed-source models, typically the most capable, achieve below 55% success on simplified reconstructed interfaces and plummet to under 9% on real OpenEMR systems. Open-source alternatives perform worse still, with dramatic performance degradation on production software. These results underscore that task completion alone is insufficient; clinical work requires reasoning about domain-specific safety dimensions beyond what general benchmarks measure.

The benchmark's paired intent and step-level goals decouple clinical reasoning from UI execution, a methodological innovation that clarifies where agent failures originate. This distinction matters for development roadmaps: improving medical knowledge doesn't help agents that struggle with UI navigation, just as better interface interaction fails when clinical judgment is flawed. The inclusion of five clinical safety dimensions moves evaluation beyond productivity metrics toward clinical validity.

For the AI industry, MedCUA-Bench establishes baseline expectations for medical applications. Healthcare organizations considering AI automation now have empirical evidence of current limitations. Future development will likely focus on domain-specific fine-tuning and safety-aware training, but the results suggest substantial engineering work remains before clinical deployment.

• →Top AI models achieve only 54.2% success on reconstructed clinical interfaces, dropping below 9% on real medical software systems
• →Open-source agents significantly underperform closed-source models, averaging 2.5% success with the best reaching just 16.2%
• →The benchmark separates clinical reasoning from UI execution through paired intent and step-level goals, revealing multifaceted agent limitations
• →Current AI agents lack readiness for clinical deployment despite general task-automation capabilities in non-medical domains
• →MedCUA-Bench establishes reproducible medical AI evaluation criteria incorporating safety dimensions beyond traditional task completion metrics