Are LLMs Ready to Assist Physicians? PhysAssistBench for Interactive Doctor-Patient-EHR Assistance
Researchers introduce PhysAssistBench, a new evaluation framework for testing large language models in real-world clinical settings where physicians, patients, and electronic health records interact simultaneously. The benchmark reveals that current leading LLMs struggle with coordinating medical knowledge, patient communication, and precise system interactions together, exposing a critical gap between isolated capability improvements and practical clinical assistance.
PhysAssistBench addresses a fundamental disconnect in medical AI evaluation. While previous benchmarks test LLM performance on narrow tasks—answering clinical questions, navigating EHR systems, or communicating with patients—real physician assistance demands seamless coordination across all three domains simultaneously. This research exposes that isolated improvements in any single capability don't translate to reliable clinical performance when models must juggle underspecified physician requests, ambiguous patient descriptions, and precise technical requirements.
The benchmark's construction from real MIMIC-IV cases with physician validation adds credibility often missing from synthetic evaluations. The scalable pipeline generating interactive agentic patients transforms static medical records into dynamic multi-turn scenarios, creating a more realistic testing environment than static question-answering datasets. This methodological rigor matters because clinical AI failures carry genuine consequences—a model performing well on isolated knowledge tests but failing at coordinated task execution could dangerously mislead practitioners into overconfidence.
For the AI and healthcare industry, this research signals that the path to clinical LLM deployment requires rethinking development priorities. Rather than chasing benchmark scores in isolated domains, teams building clinical assistants must focus on integration and robustness across interconnected capabilities. The finding that leading models remain unreliable suggests significant engineering work lies ahead before medical LLMs can safely assume meaningful clinical roles, even in advisory capacities.
The bilingual evaluation set indicates growing international recognition of this problem. Stakeholders should monitor whether major LLM developers adopt PhysAssistBench standards, as widespread adoption could become the de facto metric for clinical readiness and accelerate standardized improvements across the sector.
- →Current LLMs fail at coordinating medical knowledge, patient communication, and EHR system interaction simultaneously despite excelling at isolated tasks
- →PhysAssistBench provides a validated benchmark built from real clinical cases, offering more realistic evaluation than synthetic datasets
- →The research demonstrates that clinical AI readiness requires integration across multiple capabilities, not incremental gains in single domains
- →Physician-validated evaluation frameworks are becoming essential for establishing clinical LLM credibility and safety standards
- →Significant engineering challenges remain before medical LLMs can reliably assist physicians in real-world clinical settings