Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models

Researchers evaluated domain-specific fine-tuning of vision-language models (VLMs) on medical imaging tasks and found that performance degrades significantly with task complexity, with medical fine-tuning providing no consistent advantage. The study reveals that these models exhibit fragility and high sensitivity to prompt variations, questioning the reliability of VLMs for high-stakes medical applications.

The research challenges a widespread assumption in AI development: that domain-specific fine-tuning automatically improves model performance in specialized fields. By comparing paired models (LLaVA vs. LLaVA-Med and Gemma vs. MedGemma) across medical imaging tasks of escalating difficulty, the study demonstrates a critical limitation in current vision-language model architectures. Performance collapses toward random guessing as task complexity increases, suggesting these models lack genuine clinical reasoning capabilities and may only recognize superficial visual patterns.

The finding that medical fine-tuning provides no consistent advantage contradicts the conventional wisdom driving significant investment in specialized AI model development. This reflects a broader challenge in machine learning: fine-tuning on domain-specific data doesn't guarantee deeper understanding or robust generalization. The extreme sensitivity to prompt formulation compounds the problem—minor wording changes cause dramatic accuracy swings and variable refusal rates, indicating unstable learned representations rather than internalized medical knowledge.

The introduction of a description-based pipeline reveals an important insight: having models generate intermediate descriptions for text-only models to analyze recovers only marginal additional signal, bounded by underlying task difficulty. This suggests failures stem from both inadequate visual encoding and weak downstream reasoning pathways. For healthcare organizations and AI developers investing heavily in medical VLMs, these findings signal that current approaches may not justify clinical deployment without substantial architectural improvements.

The implications extend beyond medical imaging. If domain-specific fine-tuning fails to produce reliable improvements in well-defined technical domains like medical imaging, similar fragility likely exists in other specialized applications where decision accuracy directly impacts safety and outcomes.

• →Medical fine-tuning of vision-language models provides no consistent performance advantage over base models across imaging classification tasks.
• →Model accuracy degrades toward random levels as task difficulty increases, indicating lack of genuine clinical reasoning capacity.
• →Performance is highly sensitive to minor prompt variations, revealing unstable learned representations rather than robust medical knowledge.
• →Failures originate from both weak visual embeddings and inadequate downstream reasoning in medical VLM architectures.
• →Domain-specific fine-tuning may not reliably improve vision-language models in high-stakes specialized applications as currently designed.