Automatic Extraction of Structured Information from Brain MRI Reports Using an Open-Weight Large Language Model

Researchers evaluated LLaMA 3.1, an open-weight large language model, for extracting structured information from Dutch brain MRI reports. The model achieved high accuracy (80-96%) on visual rating scores and detection tasks, with few-shot prompting further improving performance on numerical variables, demonstrating practical viability for automated medical data extraction in radiology.

This study addresses a critical bottleneck in medical research: the manual extraction of structured data from unstructured clinical reports. By validating LLaMA 3.1's capability on 947 Dutch neuroradiology reports, the researchers demonstrate that open-weight LLMs can reliably automate this labor-intensive process without requiring specialized medical fine-tuning. The high performance on visual rating scales (87-96% accuracy) and mention detection (82-93% accuracy) suggests these models understand complex clinical terminology and radiological concepts across languages.

The research emerges as healthcare systems worldwide grapple with digitization and data standardization. Previous studies on LLMs in radiology have primarily focused on English-language reports or proprietary models, making this open-source, multilingual validation particularly valuable for international medical institutions. The finding that English translation yields comparable results to Dutch-native processing broadens accessibility, though challenges remain with precise numerical quantification—a critical limitation for clinical metrics.

For healthcare AI developers and medical institutions, this work validates open-source alternatives to expensive proprietary solutions, reducing deployment costs and increasing transparency. The few-shot prompting improvements (80% to 92% accuracy for microbleed counting) suggest that modest contextual examples substantially boost performance, making implementation pragmatic. However, the persistent weakness in location-specific variables indicates these models require structural improvements or hybrid human-AI approaches for complex spatial reasoning.

Looking forward, institutions implementing automated report processing should monitor refinements in open-weight models and benchmark performance against domain-specific baselines. The study's methodology—rigorous evaluation across multiple report splits with inter-rater reliability assessment—establishes standards for validating clinical LLM applications.

• →LLaMA 3.1 achieved 87-96% accuracy on visual rating scales in brain MRI reports, demonstrating strong capability for standard neuroradiology assessments.
• →Few-shot prompting significantly improved numerical variable extraction, boosting microbleed counting accuracy from 80% to 92%.
• →Open-weight models show comparable performance in Dutch and English-translated reports, enabling broader international deployment.
• →Challenges persist with location-specific variables and precise numerical quantification, requiring additional refinement or human oversight.
• →This validation supports cost-effective automation of clinical data extraction in healthcare systems using openly available models.