The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs

Researchers developed a framework separating language proficiency from cultural knowledge access in large language models across 13 locales and 80 models. The study reveals that while English outperforms local languages on culture-agnostic questions, local languages consistently show advantages for accessing culture-specific knowledge once proficiency gaps are controlled for. This finding challenges the assumption that weaker local-language LLM performance indicates weaker cultural knowledge.

This research addresses a critical gap in understanding how large language models handle culturally grounded information across languages. Previous evaluations conflated general language ability with domain-specific knowledge access, masking important patterns about where cultural information actually resides in these models. By employing item response theory to separate proficiency from knowledge access, the researchers uncovered a counterintuitive finding: local languages provide better access to local cultural knowledge despite often showing lower raw accuracy scores.

The study's methodology is particularly valuable because it uses real-world cultural questions from regional benchmarks rather than artificial parallel templates. This approach captures how cultural knowledge naturally appears in different linguistic contexts. The consistent pattern across diverse locales and model architectures—including frontier and regionally-aligned models—suggests this is a fundamental property of how LLMs internalize and organize information.

For developers building LLM applications in non-English markets, this research has direct implications. Raw accuracy metrics may be misleading guides for cultural question answering; models performing worse in local languages might still access cultural knowledge more effectively. Organizations deploying LLMs for localized customer service, educational content, or cultural applications should reconsider evaluation strategies. The finding that language-adapted and regionally-aligned models show clearer local-language advantages suggests that fine-tuning approaches can amplify cultural knowledge accessibility.

Future research should explore whether this pattern holds across model sizes, whether specific fine-tuning approaches can better leverage local-language cultural knowledge, and how to design evaluation frameworks that measure cultural accuracy independently of language proficiency.

• →Local languages show consistent advantages for accessing culture-specific knowledge once language proficiency is controlled for using item response theory.
• →Raw accuracy metrics mask the true performance of local languages because they conflate general language proficiency with cultural knowledge access.
• →English dominates on culture-agnostic questions but this reflects stronger English proficiency rather than superior cultural knowledge distribution.
• →Frontier and regionally-aligned models show clearer local-language advantages, suggesting fine-tuning can amplify culturally-specific knowledge.
• →Current LLM evaluation practices are inadequate for assessing cultural knowledge; new frameworks must separate proficiency from localized information access.