VeriSim: A Configurable Framework for Evaluating Medical AI Under Realistic Patient Noise

Researchers introduce VeriSim, an open-source framework that tests medical AI systems by injecting realistic patient communication barriers—such as memory gaps and health literacy limitations—into clinical simulations. Testing across seven LLMs reveals significant performance degradation (15-25% accuracy drop), with smaller models suffering 40% greater decline than larger ones, exposing a critical gap between standardized benchmarks and real-world clinical robustness.

VeriSim addresses a fundamental blind spot in medical AI evaluation: the assumption that patients communicate clearly and completely. Real clinical encounters involve patients who forget details, misunderstand medical concepts, withhold information due to stigma, or experience anxiety that impairs communication. The framework operationalizes six noise dimensions grounded in peer-reviewed medical literature, creating a more authentic testing environment than existing benchmarks.

This research exposes a critical Sim-to-Real gap in medical AI development. While LLMs demonstrate impressive performance on standardized medical QA datasets, these controlled environments fail to replicate actual patient interactions. The 15-25% accuracy degradation under realistic noise suggests current models may be poorly calibrated for deployment in clinical settings. Particularly concerning is the finding that medical fine-tuning on standard corpora provides limited robustness improvements, indicating that current training approaches may not address communication barriers effectively.

The performance differential between model sizes has important implications for healthcare accessibility. Smaller models (7B parameters) degrade 40% more than larger counterparts, meaning resource-constrained healthcare systems adopting smaller models for cost efficiency may deploy systems significantly less reliable than benchmarks suggest. Board-certified clinician validation with strong inter-annotator agreement (kappa > 0.80) establishes VeriSim's credibility as a rigorous evaluation methodology.

The release as open-source infrastructure creates an opportunity for the AI research community to develop more robust training methodologies. Future work should focus on developing models explicitly trained to handle patient communication variability and uncertainty, rather than optimizing for clean-data performance. Healthcare organizations evaluating AI systems should demand evaluation under realistic patient noise rather than relying on benchmark scores alone.

• →Medical LLMs show 15-25% accuracy degradation when tested with realistic patient communication barriers, revealing a critical Sim-to-Real gap
• →Smaller 7B-parameter models degrade 40% more than 70B+ models under patient noise, creating accessibility concerns for resource-constrained healthcare systems
• →Current medical fine-tuning approaches provide limited robustness against patient communication noise, indicating training methodologies need fundamental rethinking
• →VeriSim's clinician-validated framework with kappa > 0.80 establishes a rigorous open-source testbed for evaluating clinical AI robustness
• →Healthcare AI procurement should shift from benchmark-based evaluation to realistic patient interaction testing to ensure deployment safety