Do LLMs Build World Models From Text? A Multilingual Diagnostic of Spatial Reasoning

Researchers introduced MentalMap, a multilingual benchmark testing whether large language models can build spatial world models from text alone. The study found a universal performance cliff at reasoning level L3 across all tested models and languages, where models fail to maintain spatial reasoning accuracy despite strong baseline performance, suggesting fundamental text-only working memory constraints rather than architectural limitations.

This research reveals a critical limitation in how current large language models process and reason about spatial information. The MentalMap benchmark systematically demonstrates that LLMs struggle with viewpoint-dependent reasoning—a fundamental aspect of human spatial cognition—regardless of model scale, architecture, or language. The universal L3 cliff observed across thirteen different models suggests this is not an easily fixable engineering problem but rather reflects inherent constraints in text-only processing.

The multilingual dimension of this study strengthens its findings significantly. By testing eight typologically diverse languages plus structured text, the researchers show that spatial reasoning failures are not language-specific quirks but fundamental limitations in how LLMs construct mental models from sequential text. The fact that human subjects exhibit identical failure patterns under identical conditions provides compelling evidence that the bottleneck stems from working memory constraints inherent to pure-text modalities, not from LLM-specific architectural choices.

For AI development, these findings have substantial implications. Current LLM applications relying on spatial reasoning—from robotics navigation to scene understanding—operate below an accuracy threshold that limits real-world deployment. The research suggests that simple scaling and prompting strategies alone won't overcome this cliff. Instead, multimodal approaches integrating visual information or memory-augmented systems with scratchpad mechanisms represent necessary directions. This work effectively identifies a quantifiable, reproducible limitation that the field must address through architectural innovation rather than incremental improvements to existing text-only approaches.

• →All tested LLMs exhibit a universal performance cliff at spatial reasoning level L3, losing half their accuracy when advancing beyond atomic spatial facts.
• →The spatial reasoning limitation persists across model scales, families, and languages, indicating a fundamental rather than incidental problem.
• →Human subjects replicate identical failure patterns under pure-text conditions, suggesting text-only working memory constraints rather than LLM-specific architecture flaws.
• →Multimodal integration and memory-augmented reasoning with scratchpads are necessary directions to overcome pure-text spatial reasoning bottlenecks.
• →The MentalMap benchmark provides a standardized multilingual framework for systematically diagnosing and measuring spatial world-modeling capabilities across AI systems.