PROTON: Prototype-Based Test-Time Online OOD Detection for Medical VLMs

Researchers introduce PROTON, a lightweight post-hoc module that improves out-of-distribution detection in medical vision-language models by combining prototype-based distance metrics with traditional scoring methods. The approach achieves significant performance gains across multiple distribution shift types without requiring model retraining or labeled data.

Medical vision-language models face a critical deployment challenge: reliably identifying when input data falls outside their training distribution. Traditional static scoring methods like Maximum Concept Matching (MCM) exhibit severe performance degradation under specific shift types, achieving only 42.4% AUROC on covariate shifts despite 76.4% accuracy on far-OOD inputs. This inconsistency stems from a fundamental structural mismatch where covariate-shifted data appears indistinguishable from in-distribution samples in softmax probability space but occupies distinct regions in VLM embedding layers.

PROTON addresses this gap through an elegant post-hoc design that maintains a dynamic prototype bank from high-confidence predictions during deployment. The system adaptively weights prototype-distance signals against MCM scores using stream-level variance statistics, effectively exploiting previously unutilized embedding-space information. The approach requires no architectural modifications, training data, or manual prompt engineering—critical advantages for practical clinical adoption where computational resources and ground-truth labels are scarce.

On the FLAIR+FIVES ophthalmology benchmark, PROTON delivers substantial improvements: +23.9 AUROC on covariate shifts, +8.8 on semantic shifts, and +8.1 on far-OOD scenarios. More importantly, it represents the first zero-shot method achieving consistent gains across all three distribution shift categories. This breakthrough has immediate implications for medical AI deployment, where failure modes from distribution shifts directly impact diagnostic accuracy and patient safety. The availability of open-source code accelerates adoption across healthcare institutions.

• →PROTON improves OOD detection by fusing VLM embedding-space signals with traditional softmax-based scoring, achieving +23.9 AUROC on covariate shifts.
• →The post-hoc design requires no model retraining, labeled data, or prompt engineering, reducing deployment friction for medical institutions.
• →First zero-shot OOD detection method to improve performance across all three distribution shift types simultaneously.
• →Lightweight prototype banking approach operates at test-time with minimal computational overhead.
• →Open-source implementation enables rapid integration into existing medical VLM pipelines.