XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity

Researchers introduce XL-SafetyBench, a comprehensive safety evaluation framework for large language models across 10 country-language pairs with 5,500 test cases. The study reveals that frontier LLMs show decoupled jailbreak robustness and cultural awareness, while local models often exhibit apparent safety driven by generation failure rather than genuine alignment.

XL-SafetyBench addresses a critical gap in LLM safety evaluation: the predominance of English-centric benchmarks that fail to capture country-specific harms and cultural sensitivities. Traditional safety assessments rely heavily on translation and universal harm definitions, overlooking nuanced regional concerns embedded in language and culture. This research introduces a methodologically rigorous approach combining LLM-assisted discovery with dual native-speaker annotation, establishing three distinct metrics—Attack Success Rate, Neutral-Safe Rate, and Cultural Sensitivity Rate—rather than collapsing safety into a single composite score.

The findings carry significant implications for AI developers and deployment strategies. The decoupling of jailbreak robustness from cultural awareness among frontier models suggests that safety training and cultural alignment require distinct interventions. More troubling is the near-perfect negative correlation between attack success rates and neutral-safe rates among local models, indicating that apparent safety often reflects comprehension failures or generation limitations rather than principled refusal. This distinction matters critically for real-world deployment, where false confidence in safety could mask underlying vulnerabilities.

For the broader AI industry, XL-SafetyBench establishes a benchmark for more rigorous, geographically inclusive safety evaluation. As LLMs proliferate across non-English markets, developers must recognize that safety is not universally transferable and that localized models require independent validation against region-specific harms. The framework's multi-stage pipeline becomes a template for future research. For investors and stakeholders evaluating AI safety claims, the research highlights that marketing composite safety scores obscures critical performance variations—a consideration for due diligence on AI systems deployed globally.

• →XL-SafetyBench introduces a 5,500-case multilingual safety benchmark addressing English-centric evaluation bias in current LLM testing frameworks.
• →Frontier models show decoupled jailbreak robustness and cultural awareness, meaning a single safety score masks significant per-axis performance variation.
• →Local LLMs demonstrate a -0.81 correlation between attack success rates and neutral-safe rates, indicating apparent safety often reflects generation failure rather than genuine alignment.
• →The framework introduces three distinct metrics—ASR, NSR, and CSR—to differentiate principled refusal from comprehension failures across cultures.
• →Results reveal that safety training and cultural sensitivity require distinct interventions, not interchangeable approaches.