ClinicalMC: A Benchmark for Multi-Course Clinical Decision-Making with Large Language Models

Researchers introduce ClinicalMC, a benchmark dataset designed to evaluate how large language models perform in complex, multi-stage clinical decision-making scenarios where patient conditions evolve over time. The benchmark includes 7,079 samples across English and Chinese datasets with a multi-agent evaluation framework, testing closed-source, open-source, and medical-specialized LLMs.

ClinicalMC addresses a critical gap in AI healthcare evaluation by shifting focus from single-instance clinical assessments to realistic multi-course patient journeys. Traditional benchmarks have isolated LLM performance to individual diagnostic moments, failing to capture how models handle evolving patient conditions, treatment responses, and sequential decision-making—core requirements for actual clinical deployment. This benchmark matters because healthcare AI adoption increasingly demands models that can track patient progression across multiple visits and adjust recommendations based on cumulative clinical data.

The research reflects growing maturity in AI healthcare evaluation. As LLMs integrate deeper into clinical workflows, standardized testing becomes essential for regulatory approval and clinical trust. The inclusion of both English (5,804 samples) and Chinese (1,275 samples) datasets acknowledges different healthcare systems and language-specific clinical reasoning patterns. The multi-agent framework—simulating patient, examiner, and doctor roles—creates more realistic interaction scenarios than static question-answer formats.

For the AI healthcare industry, this benchmark establishes evaluation standards that will influence future model development and deployment decisions. Organizations deploying medical LLMs will face pressure to validate performance against such rigorous multi-course scenarios. The competitive testing of closed-source models (GPT-style), open-source alternatives (DeepSeek), and specialized medical LLMs (HuatuoGPT) provides transparent performance comparisons that inform procurement and development priorities.

The framework's significance extends to regulatory bodies developing AI healthcare guidelines. Standardized benchmarks accelerate approval processes by providing objective performance metrics. Future work will likely expand such benchmarks to additional languages and more complex clinical scenarios, establishing ClinicalMC as foundational infrastructure for medical AI validation.

• →ClinicalMC introduces first multi-course clinical decision-making benchmark with 7,079 samples spanning triage through discharge stages
• →Multi-agent evaluation framework simulates realistic clinical interactions beyond single-turn question-answer assessments
• →English dataset shows patients average 5.11 clinical courses versus 3.42 for Chinese dataset, revealing healthcare system differences
• →Competitive evaluation of closed-source, open-source, and medical-specialized LLMs provides transparent performance comparison
• →Benchmark addresses critical gap in evaluating LLM performance on evolving patient conditions across multiple clinical encounters