The Open-Box Fallacy: Why AI Deployment Needs a Calibrated Verification Regime

Researchers propose replacing mechanistic interpretability requirements with 'calibrated verification' for AI deployment in sensitive domains like healthcare and criminal justice. The framework emphasizes domain-specific authorization, independent monitoring, and accountability mechanisms rather than demanding full model explainability, citing evidence that understanding model internals doesn't ensure safe real-world outcomes.

The paper addresses a critical governance gap in AI deployment by challenging the prevailing assumption that understanding how AI models work internally is necessary for safe authorization. Current regulatory approaches often demand mechanistic interpretability as a gating mechanism, but this study reveals a fundamental disconnect: researchers achieved a 53-percentage-point gap between understanding internal representations and producing output corrections, indicating that technical comprehension alone fails to guarantee responsible deployment. This distinction mirrors how societies have historically governed opaque expertise in medicine, law, and finance—through credentials, monitoring, liability structures, and revocation rights rather than requiring practitioners to explain every cognitive process. The research notes that only 9% of FDA-approved AI/ML medical devices included prospective post-market surveillance, exposing a verification vacuum in current regulatory frameworks. The proposed Verification Coverage standard shifts focus from internal mechanisms to six measurable components addressing scope, checkability, monitoring, accountability, contestability, and revocability. This approach better reflects operational reality: AI capabilities vary dramatically across similar tasks, making blanket model authorization impossible. For industry stakeholders, the framework suggests future regulatory requirements will emphasize deployment-specific credentials and continuous monitoring rather than interpretability demands that may be technically infeasible or operationally irrelevant. This could accelerate AI adoption in regulated sectors by replacing the interpretability bottleneck with more practical, outcome-focused governance structures.

• →Mechanistic interpretability of AI models does not correlate reliably with safe deployment decisions or output correction.
• →Authorization frameworks should be domain-specific and use-case-scoped rather than model-scoped, as capability varies across similar tasks.
• →Calibrated verification emphasizing monitoring, accountability, and revocation is more effective than demanding full model explainability.
• →Current FDA-approved AI/ML medical devices lack adequate post-market surveillance, creating governance failures in sensitive domains.
• →The proposed Verification Coverage standard provides a measurable, reportable metric for model cards and regulatory disclosures.