Evaluating Patient Safety Risks in Generative AI: Development and Validation of a FMECA Framework for Generated Clinical Content

Researchers developed and validated the first FMECA (Failure Mode, Effects, and Criticality Analysis) framework to systematically assess patient safety risks in clinical summaries generated by large language models. Testing with GPT-OSS 120B on real hospital discharge summaries demonstrated moderate-to-substantial inter-rater agreement and identified 14 distinct failure modes, establishing a reproducible methodology for evaluating AI-generated clinical content safety.

This research addresses a critical gap in AI healthcare deployment: the absence of structured safety assessment methods for LLM-generated clinical content. As healthcare systems increasingly adopt generative AI for documentation and summarization tasks, understanding failure modes and their clinical consequences becomes essential for patient protection. The study's development of a FMECA framework—adapted from traditional engineering risk assessment—represents a methodological advance in proactive AI safety evaluation rather than reactive incident response.

The research emerged from growing recognition that LLMs, despite impressive performance metrics, introduce novel failure modes in clinical contexts where errors directly impact patient outcomes. The interdisciplinary approach, combining clinical expertise with formal risk assessment methodology, reflects industry maturation toward evidence-based AI governance. The framework's 14 failure mode categories capture clinically relevant risks that generic AI safety evaluations might miss.

For healthcare organizations and AI developers, this framework provides a standardized approach to implementing responsible AI practices in clinical settings. The moderate-to-substantial inter-rater agreement validates the framework's reliability, though some refinement remains necessary. Strong usability ratings (79.2/100 SUS) suggest practical applicability. The methodology extends beyond clinical summarization, potentially applicable to other LLM-generated medical content including clinical notes, differential diagnoses, and treatment recommendations.

The validation work establishes baseline expectations for similar AI safety frameworks across healthcare domains. Organizations deploying clinical LLMs face regulatory pressure to demonstrate safety governance; this framework provides structured evidence. Future work should expand validation across diverse LLM architectures, clinical specialties, and real-world deployment scenarios to strengthen generalizability and establish clinical implementation standards.

• →First validated FMECA framework specifically designed for assessing patient safety risks in LLM-generated clinical content.
• →Framework identified 14 distinct failure modes with moderate-to-substantial inter-rater agreement, demonstrating reproducibility and clinical utility.
• →Strong usability ratings (79.2/100) indicate the framework is practical for clinical implementation and evaluation.
• →Addresses critical gap in healthcare AI governance by providing proactive risk assessment methodology rather than reactive incident response.
• →Establishes methodological foundation for broader AI safety assessment across clinical applications and potentially other high-stakes domains.