CardioLens: Revealing the Clinical Reality Gap of MLLMs via Multi-Sequence Cardiac MRI Evaluations

Researchers introduce CardioLens, a rigorous evaluation framework revealing that state-of-the-art multimodal large language models (MLLMs) perform poorly at clinical cardiac MRI interpretation despite strong public benchmark results. The study demonstrates a significant gap between theoretical capabilities and real-world clinical applicability, with models failing to integrate distributed evidence across imaging sequences and temporal phases.

CardioLens exposes a critical vulnerability in the current AI medical imaging landscape: the disconnect between benchmark performance and clinical readiness. While MLLMs consistently achieve high scores on public medical datasets, this research demonstrates that real-world cardiac MRI interpretation requires capabilities these models fundamentally lack. The evaluation framework, constructed from 473,896 slices of private hospital data across multiple cardiac imaging modalities, represents a methodologically rigorous approach to testing AI reliability in clinical contexts.

This work builds on growing concerns about AI overfitting to benchmark datasets. Previous research highlighted similar performance gaps in other medical domains, but CardioLens provides concrete evidence specific to cardiovascular imaging—a domain where diagnostic errors carry direct patient safety implications. The study's design eliminates common evaluation shortcuts; researchers implemented a leakage-resistant pipeline and tested whether input selection strategies could artificially inflate results, finding minimal performance variance.

For medical AI developers and healthcare institutions, CardioLens signals that deployment decisions require substantially more rigorous validation than current public benchmarks provide. The finding that explicit reasoning prompts often increase model conservatism rather than improve diagnostic accuracy suggests fundamental architectural limitations in current MLLMs for integrating complex spatial and temporal information. Healthcare providers evaluating AI implementation should treat published benchmark results with significant skepticism.

The research establishes a replicable methodology for constructing clinically grounded evaluation frameworks, potentially spurring similar assessments across other medical specialties. Future MLLM development will likely need to address the category-collapse failure mode and improve integration of distributed evidence across imaging sequences before achieving clinical reliability.

• →MLLMs show substantial performance degradation on real cardiac MRI tasks compared to public benchmark results, revealing a clinical reality gap
• →Models exhibit category-collapse failure, defaulting to frequent abnormal diagnoses rather than distinguishing clinically distinct findings
• →Input selection strategies and explicit reasoning prompts provide minimal performance improvements, suggesting architectural limitations rather than input construction issues
• →CardioLens establishes a leakage-resistant evaluation methodology using 473,896 slices across multiple cardiac imaging modalities with verified QA pairs
• →Current MLLMs lack capability to integrate distributed evidence across imaging sequences, views, and temporal phases required for clinical cardiac interpretation