Prompt Sensitivity and Answer Consistency of Small Open-Source Large Language Models on Clinical Question Answering: Implications for Low-Resource Healthcare Deployment

Research evaluated five small open-source language models on clinical question answering, finding that high consistency doesn't guarantee accuracy - models can be reliably wrong. Llama 3.2 showed the best balance of accuracy and reliability, while roleplay prompts consistently reduced performance across all models.

• →Small open-source AI models show dangerous inconsistency in medical applications, with high consistency not correlating with correctness
• →Llama 3.2 demonstrated the strongest balance of accuracy and reliability for low-resource healthcare deployment
• →Roleplay prompts consistently reduced accuracy across all models and should be avoided in healthcare applications
• →Domain-specific pretraining alone is insufficient for reliable clinical AI performance without instruction tuning
• →Safe clinical AI deployment requires joint evaluation of consistency, accuracy, and instruction adherence