Evaluating Reliability Gaps in Large Language Model Safety via Repeated Prompt Sampling

Researchers introduce Accelerated Prompt Stress Testing (APST), a new evaluation framework that reveals safety vulnerabilities in large language models through repeated prompt sampling rather than traditional broad benchmarks. The study finds that models appearing equally safe in conventional testing show significant reliability differences when repeatedly queried, indicating current safety benchmarks may mask operational risks in deployed systems.

Current LLM safety evaluation frameworks like HELM and AIR-BENCH prioritize breadth—testing models across diverse tasks—but miss a critical vulnerability class: consistency failures under repeated use. This gap matters because production systems encounter the same prompts repeatedly, yet existing benchmarks typically sample responses only once or twice per prompt. APST addresses this by treating LLM failures as stochastic events, using statistical modeling to estimate failure probabilities across temperature settings and prompt variations. The research reveals that models performing identically in shallow evaluations (three or fewer samples) diverge substantially when tested at scale, with failure rates varying meaningfully across inference conditions. This finding has significant implications for high-stakes deployments like healthcare, finance, or safety-critical applications where consistency is non-negotiable. Organizations relying on single-sample benchmarks to validate model safety may unknowingly deploy systems with unacceptable operational failure rates. The work bridges reliability engineering principles—traditionally applied to hardware—into AI safety evaluation, providing quantitative risk estimation rather than binary pass-fail judgments. Looking ahead, the research suggests safety certification processes need fundamental redesign to incorporate repeated-use testing protocols. Model developers must adopt depth-oriented evaluation alongside breadth metrics, and procurement standards should demand empirical failure probability data. As LLMs proliferate in critical infrastructure, APST-style testing could become table stakes for responsible deployment.

• →Traditional LLM safety benchmarks mask reliability failures that emerge under repeated use in production systems
• →Statistical modeling of failure probabilities reveals substantial performance differences invisible in conventional single-sample testing
• →Temperature variation and prompt perturbation expose latent failure modes like hallucinations and inconsistent refusals
• →Models rated equally safe by existing benchmarks show dramatically different operational risk profiles under stress testing
• →Depth-oriented evaluation frameworks may become essential for safety certification in high-stakes AI deployments