Policy-Grounded Safety Evaluation of 20 Large Language Models

Researchers introduced Aymara AI, a programmatic platform for safety evaluation of large language models, testing 20 commercially available LLMs across 10 safety domains. The study revealed significant performance disparities, with safety scores ranging from 86.2% to 52.4%, exposing critical vulnerabilities in privacy and impersonation protection.

The Aymara AI study addresses a critical gap in responsible AI development by providing the first systematic, policy-grounded evaluation framework for large language models at scale. As LLMs become embedded in enterprise applications, healthcare, finance, and government systems, the ability to rigorously assess safety risks has become essential infrastructure. The platform's innovation lies in its methodology—converting abstract safety policies into concrete adversarial test cases and using AI-validated scoring against human benchmarks—enabling reproducible, customizable evaluations without relying solely on manual review.

The empirical findings are sobering and illuminate why LLM safety remains fragmentary. Models excel at preventing obvious harms like misinformation (95.7% mean safety), where guardrails are mature and well-understood, but catastrophically fail at nuanced challenges like privacy and impersonation (24.3% mean safety). This divergence suggests that current safety approaches depend heavily on well-trodden domains while leaving novel attack surfaces exposed.

For AI developers and enterprises, these results underscore the inadequacy of blanket safety certifications. Organizations deploying LLMs for sensitive tasks—customer service, data handling, identity verification—cannot rely on generic model ratings. The demonstrated variance across models and domains requires deployment-specific safety validation.

Looking forward, Aymara AI establishes a precedent for systematic safety evaluation that regulators and enterprise procurement teams are likely to demand. The framework's customizability positions it as foundational infrastructure for responsible AI governance, though widespread adoption depends on whether organizations treat safety evaluation as a compliance checkbox or genuine operational priority.

• →Safety scores across 20 LLMs ranged from 86.2% to 52.4%, revealing inconsistent protection across models and domains.
• →Models perform significantly better on established safety domains like misinformation (95.7%) versus complex areas like privacy and impersonation (24.3%).
• →Aymara AI's policy-to-adversarial-prompts methodology enables scalable, reproducible, and customizable LLM safety evaluation.
• →The study establishes empirical evidence that generic LLM safety certifications are insufficient for domain-specific deployment.
• →Regulatory and enterprise focus on LLM safety evaluation will likely increase as deployment in sensitive applications expands.