Translating Inference-Time Control to Radiology Vision-Language Models: Activation Steering for Pneumonia Classification on Chest X-rays

Researchers evaluated Contrastive Activation Addition (CAA), an inference-time technique, to improve pneumonia classification in frozen chest X-ray vision-language models without fine-tuning. Testing three medical VLMs on a pneumonia benchmark, the team achieved meaningful F1 score improvements in one model through activation steering, suggesting this lightweight approach could adapt medical AI systems post-deployment.

This research addresses a practical challenge in medical AI deployment: improving model performance without retraining or updating weights. Contrastive Activation Addition represents an emerging class of inference-time techniques that manipulate internal model representations to steer outputs toward desired behaviors. The study's methodological rigor—including reverse-vector controls, bootstrap confidence intervals, and multi-model evaluation—demonstrates mature scientific practice in validating AI safety interventions.

The broader context reflects growing interest in post-hoc model adaptation as computational demands and regulatory constraints make full retraining expensive or impractical. Medical imaging represents an ideal proving ground because diagnostic performance is measurable, stakes are clear, and models must operate reliably in resource-constrained clinical environments. The use of frozen, publicly available VLMs (MedGemma, NV-Reason-CXR, CheXOne) highlights how researchers leverage existing foundation models rather than building custom systems.

Results reveal both promise and limitations. NV-Reason-CXR-3B showed substantial gains (F1 improvement from 0.77 to 0.87), while other models showed marginal or statistically uncertain improvements. This variability suggests activation steering effectiveness depends heavily on model architecture and training methodology—not a universal solution. For the medical AI industry, this implies practitioners cannot assume inference-time steering will work across all systems; empirical validation remains essential before clinical deployment.

The findings warrant continued investigation into why some models respond robustly to steering while others resist. Future work should explore whether steering techniques transfer across domains, whether they maintain performance under distribution shift, and whether clinicians can meaningfully interact with steering parameters in practice.

• →Contrastive Activation Addition improved pneumonia classification F1 scores in one of three tested medical VLMs without retraining or weight updates.
• →NV-Reason-CXR-3B showed the strongest gains, with calibrated F1 increasing from 0.77 to 0.87 using image-conditioned steering vectors.
• →Activation steering substantially altered prediction score distributions and operating characteristics, offering a lightweight adaptation approach for deployed models.
• →Model responsiveness to steering varied significantly across architectures, suggesting effectiveness is not universal and requires empirical validation per system.
• →The technique enables post-deployment performance tuning in resource-constrained medical environments without computational costs of full retraining.