Towards AI epidemiology: a measurement standardisation framework for prospective risk detection

Researchers propose a measurement standardization framework for detecting risks in deployed AI systems through structured expert-AI interaction analysis, without requiring access to model internals. The framework aims to establish reliable alignment scoring methodologies that could enable institutional monitoring of AI behavior and support epidemiological studies of AI-related outcomes in professional settings.

This paper addresses a fundamental challenge in AI governance: how to systematically evaluate and monitor AI system behavior in real-world deployments when organizations lack visibility into model internals. The proposed framework converts expert-AI interactions into standardized, comparable metrics—a necessary step toward institutional oversight of AI systems operating across critical domains.

The research builds on growing recognition that effective AI risk management requires measurement infrastructure comparable to public health epidemiology. Just as epidemiologists track disease patterns through standardized data collection, this framework proposes tracking AI alignment and safety through structured assessment protocols. The staged research approach reflects scientific rigor by separating measurement validation (the focus here) from subsequent claims about governance applications and epidemiological associations.

For organizations deploying large language models in regulated professional settings—healthcare, finance, law—this framework offers practical value. Real-time alignment scoring could provide deployment-phase signals, while institutional pattern analysis across models and domains enables evidence-based decisions about AI system trust and scope. The statistical rigor specified (paired bootstrap inference, DeLong's test, non-inferiority margins) demonstrates sophisticated methodology design.

The framework's broader significance lies in shifting AI oversight from opaque model evaluation toward behavioral monitoring in actual use contexts. This approach accommodates the reality that many organizations cannot or will not open-source their models, yet stakeholders require assurance mechanisms. Success of this research programme could establish a new field of AI safety measurement, creating standards that regulators and institutions could adopt systematically.

• →Proposed framework standardizes expert assessment of AI-system alignment without requiring access to proprietary model internals.
• →Three-tier research programme: first validates measurement reliability, second enables institutional deployment monitoring, third supports epidemiological risk studies.
• →Methodology includes defined grammar, bootstrap inference, DeLong's test, and Holm-Bonferroni correction for statistical rigor.
• →Framework addresses critical gap in AI governance by enabling institutions to monitor alignment patterns across models and domains.
• →Approach treats AI safety measurement analogously to epidemiological disease tracking, potentially establishing new oversight standards.