Cross-modal linkage risk in clinical vision-language models

Researchers discovered that vision-language models trained on paired chest X-rays and medical reports can re-link de-identified images to their original reports through embedding similarity, creating a privacy vulnerability. The team demonstrated this risk scales with model specialization and developed a differential privacy technique that reduces re-linkage by 62% while preserving diagnostic utility.

Vision-language models have become increasingly valuable in healthcare by learning associations between medical images and their corresponding reports, enabling sophisticated clinical decision support. However, this research exposes a fundamental tension in healthcare data sharing: the same learned associations that make these models useful for diagnosis create privacy risks when images and reports are deliberately separated for compliance or access-control reasons. The finding is significant because it demonstrates that de-identified medical images aren't actually de-identified when paired with a trained VLM and a database of reports—a previously underexplored vulnerability in the clinical AI pipeline.

The study's scope is substantial, evaluating over 400,000 paired examples across multiple datasets and showing that re-linkage success increases dramatically with model sophistication. This progression suggests the privacy risk isn't merely a surface-level artifact but reflects deep structural learning of image-report correspondence. The researchers' solution—applying differential privacy only to the projection heads rather than retraining entire models—is pragmatic for real-world deployment since it sidesteps computational costs while maintaining image-side utility for downstream clinical tasks.

For healthcare organizations and AI developers, this work establishes a new audit standard for multimodal medical models before deployment in regulated environments. The 62% reduction in re-linkage at scale demonstrates feasibility of privacy-preserving approaches, but the persistence of above-chance performance even after mitigation suggests this remains an open problem. As medical institutions increasingly adopt sophisticated VLMs for radiology and other specialties, privacy-aware training and evaluation protocols will become essential compliance and risk-management measures.

• →Vision-language models can re-link de-identified medical images to their original reports through embedding similarity, defeating data separation controls.
• →Re-linkage success scales systematically with model specialization, reaching 50x chance-level performance in realistic candidate pool sizes.
• →Differential privacy applied to alignment layer projections reduces re-linkage by 61.8% while maintaining clinical diagnostic performance above 79%.
• →The privacy vulnerability persists even when pathology-matched negatives remove disease-label shortcuts, indicating learning of deeper image-report correspondence.
• →Privacy-aware evaluation of multimodal medical models should become standard before clinical deployment in regulated healthcare environments.