MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinically Realistic EHR Settings

Researchers introduced MedCase-Structured, a synthetic dataset that converts unstructured clinical text into standardized HL7 FHIR format for evaluating large language models in realistic healthcare settings. The study reveals that LLMs perform significantly worse on structured clinical data than plain text, highlighting a critical gap between academic benchmarks and real-world deployment requirements.

This research addresses a fundamental disconnect in how large language models are evaluated for clinical applications. While LLMs have demonstrated strong performance on medical reasoning tasks in academic settings, the study reveals that performance degrades substantially when models encounter the structured, standardized formats actually used in hospital electronic health record systems. This gap matters because healthcare adoption depends on models functioning reliably within existing clinical infrastructure, not just answering isolated questions.

The research emerged from growing recognition that static, text-based benchmarks fail to capture deployment realities. Clinical systems operate on interoperable data standards like HL7 FHIR R4, which impose strict structural and semantic requirements. By constructing a pipeline that generates valid FHIR bundles from narrative medical cases while maintaining clinical accuracy, the authors created a more authentic evaluation environment. The achievement of 82.5% valid FHIR generation represents substantial technical progress, though the lower diagnostic accuracy on structured inputs reveals that LLMs struggle with format-constrained reasoning.

For healthcare AI development, this finding carries significant implications. Organizations building clinical decision support systems must validate performance on realistic data formats, not just curated datasets. The research suggests that model fine-tuning or architectural adaptations may be necessary for clinical deployment. Developers and healthcare institutions should reassess existing LLM evaluations against this standard, as apparent clinical competence may not translate to functional integration with hospital systems. The work establishes a reproducible methodology for bridging the evaluation-to-deployment gap, setting a foundation for more rigorous clinical AI benchmarking practices moving forward.

• →LLMs show measurably lower diagnostic accuracy on structured FHIR clinical data compared to unstructured text inputs.
• →The MedCase-Structured dataset provides a methodology for creating clinically realistic benchmarks aligned with actual healthcare system requirements.
• →Current academic evaluations of clinical AI may overstate real-world performance due to lack of deployment-congruent testing.
• →Healthcare organizations must validate AI models against standardized data formats before clinical adoption.
• →The research establishes terminology-grounded validation as a technique for reducing hallucinated medical codes in synthetic datasets.