EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs

Researchers introduce EHRBench, an automated benchmark containing nearly 1 million QA items derived from real patient electronic health records to evaluate large language models on clinical decision-making tasks. The framework combines LLM-based template generation with knowledge-base verification to assess model performance on diagnosis, treatment, and prognosis at scale while maintaining reliability.

EHRBench addresses a critical gap in AI evaluation: the lack of scalable, reliable benchmarks for assessing LLM performance on real-world clinical tasks. While LLMs show promise in healthcare applications due to their language capabilities and broad biomedical knowledge, their reliability for actual clinical decision-making remains largely unvalidated. This work matters because deploying unvetted AI systems in clinical settings carries significant risks, and clinicians need objective performance data before integration into workflows.

The research emerges from a broader trend of healthcare AI applications outpacing rigorous evaluation frameworks. Existing benchmarks often rely on curated datasets disconnected from real clinical practice or suffer from quality issues like hallucinated medical relationships. EHRBench's innovation lies in its hybrid pipeline: specialized LLMs convert actual EHR trajectories into structured templates, which are then automatically instantiated into QA items while knowledge-base verification filters unreliable content. This approach balances scale (nearly 1 million items) with quality assurance—a previously difficult trade-off.

For the industry, this benchmark enables more meaningful comparisons across 30+ LLM architectures and establishes performance baselines for clinical applications. Hospitals and healthcare organizations can use these results to make informed decisions about which models suit specific clinical tasks, while AI developers gain actionable data on capability gaps and robustness issues.

Moving forward, watch for adoption of EHRBench as a standard evaluation tool in healthcare AI research and potential refinements addressing specific clinical specialties. The framework's success may accelerate development of clinically-reliable LLM systems and influence regulatory standards for AI deployment in healthcare settings.

• →EHRBench contains 960,067 QA items grounded in real patient EHRs across diagnosis, treatment, and prognosis tasks.
• →An automated EHR-LLM-KB pipeline balances scalability with reliability by filtering hallucinated medical relationships.
• →Benchmarking 30+ LLMs reveals consistent capability trends, validating the framework's reliability for clinical decision-making evaluation.
• →The dataset enables objective performance comparison necessary for clinical AI deployment decisions by healthcare institutions.
• →Knowledge-base verification and enrichment significantly improve data quality beyond raw template instantiation.