Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural Contexts?

Researchers found that large language models experience accuracy drops of 0.3% to 5.9% when math problems are presented in unfamiliar cultural contexts, even when the underlying mathematical logic remains identical. Testing 14 models across culturally adapted variants of the GSM8K benchmark reveals that LLM mathematical reasoning is not culturally neutral, with errors stemming from both reasoning failures and calculation mistakes.

This research exposes a fundamental vulnerability in how leading LLMs process mathematical problems: their performance degrades measurably when presented with culturally unfamiliar scenarios, despite unchanged mathematical operations. The study's rigor—analyzing 18,887 instances across six geographically diverse cultural contexts (Haiti, Moldova, Pakistan, Solomon Islands, Somalia, Suriname)—demonstrates that the phenomenon is statistically significant and reproducible across multiple model architectures from major AI labs.

The findings challenge the assumption that mathematical reasoning is a culturally agnostic capability. When problems are recontextualized with unfamiliar names, foods, and places, models struggle not merely with numerical computation but with the broader reasoning patterns required to structure solutions. The 54.7% of failures attributed to mathematical reasoning errors versus 34.5% to calculation errors suggests the primary issue stems from problem comprehension and logical structuring rather than arithmetic itself.

For the AI industry, this highlights a critical training data bias: models absorb cultural context during pretraining and depend on familiar scenario framing to activate optimal reasoning pathways. The observation that Mistral performs disproportionately well on Pakistan-adapted problems due to exposure to South Asian training data reinforces this conclusion—broader training diversity directly improves cross-cultural mathematical reasoning.

Developers building AI systems for global markets should recognize that mathematical accuracy claims require cultural validation. Organizations deploying LLMs for financial modeling, scientific computation, or educational applications in non-Western contexts face real performance degradation risks. Future model development must prioritize diverse training corpora and cultural representation to achieve genuinely robust mathematical reasoning across global contexts.

• →LLM math accuracy drops 0.3-5.9% when problems embed unfamiliar cultural contexts, despite identical mathematical logic
• →Mathematical reasoning errors (54.7%) exceed calculation errors (34.5%), indicating comprehension and framing issues drive failures
• →Mistral outperforms larger models on Pakistan-adapted problems due to greater South Asian training data exposure
• →Cultural familiarity activates different reasoning pathways, proving mathematical ability in LLMs is not culturally neutral
• →Global AI deployment requires cultural validation testing to ensure robust performance across diverse user populations