EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning

Researchers introduce EHR-Complex, a large-scale benchmark with 52K tasks for evaluating AI clinical agents on real-world electronic health record analysis. Testing reveals significant limitations, with top models achieving only 62.3% accuracy and exposure of three dominant failure modes: SQL logic errors, medical code lookup failures, and semantic misunderstandings.

EHR-Complex addresses a critical gap in AI evaluation by moving beyond idealized benchmarks to test clinical agents on realistic, complex medical data reasoning. Built on MIMIC-IV's 365K patient records across 31 tables, the benchmark reflects genuine EHR complexity with an average of 31.93 SQL structural components per query, demanding longitudinal multi-table aggregation and compositional reasoning capabilities that static benchmarks typically avoid.

The research exposes a sobering reality about current AI limitations in healthcare. Despite advances in large language models, the 62.3% exact-match accuracy ceiling for top performers demonstrates that clinical reasoning remains a formidable challenge. The sharp drop in Pass@k consistency below 50% at k=4 across nearly all models reveals systemic fragility in these systems' ability to produce reliable outputs—a critical concern for any healthcare application where consistency matters.

The analysis of 3,800+ failed trajectories provides actionable intelligence for developers. SQL logic errors, medical code lookup failures, and semantic misunderstandings represent distinct technical bottlenecks that require different solutions. These aren't abstract problems but concrete obstacles preventing deployment of AI agents in real clinical workflows. Healthcare institutions relying on AI-assisted EHR analysis should recognize that current systems require substantial human oversight and verification.

Moving forward, EHR-Complex serves as a rigorous testing ground for advancing clinical AI. The benchmark's rigor and scale suggest that meaningful progress requires not just larger models but architectures specifically engineered for medical reasoning, robust medical knowledge integration, and error-resistant SQL generation. Organizations developing healthcare AI should expect this benchmark to become a standard evaluation criterion.

• →EHR-Complex benchmark reveals top AI models achieve only 62.3% accuracy on realistic clinical database reasoning tasks, indicating significant gaps in healthcare AI readiness.
• →Three dominant failure modes—SQL logic errors, medical code lookup failures, and semantic misunderstandings—represent distinct technical bottlenecks requiring targeted solutions.
• →Stochastic fragility is severe, with Pass@k consistency dropping below 50% for nearly all models at k=4, raising concerns about system reliability in clinical settings.
• →The benchmark's 52K tasks on MIMIC-IV data with 31.93 average SQL components per query reflects genuine real-world EHR complexity absent from prior benchmarks.
• →Healthcare organizations should expect current AI clinical agents require substantial human verification and oversight before safe deployment in production workflows.