Measuring What VLMs Don't Say: Validation Metrics Hide Clinical Terminology Erasure in Radiology Report Generation

Researchers identify a critical flaw in Vision-Language Model evaluation for radiology, where high benchmark scores mask models' failure to generate clinically specific terminology. They propose new metrics including Clinical Association Displacement (CAD) to measure bias and clinical signal loss across demographic groups.

• →Current VLM validation metrics hide 'template collapse' where models generate generic text instead of clinical terminology.
• →High token-overlap scores can mislead about model performance, creating a metric gaming problem.
• →Clinical Association Displacement (CAD) framework quantifies demographic bias in generated medical reports.
• →Deterministic decoding produces semantic erasure while stochastic sampling risks introducing new biases.
• →The research calls for fundamental rethinking of optimal medical report generation standards.