AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materials

Researchers introduced AtomWorld, a benchmark for evaluating how well large language models can perform spatial reasoning tasks in materials science, specifically atomic structure manipulation. The study reveals that current LLMs like Claude Opus 4.6 struggle with complex spatial operations, achieving success rates below 12% for rotation tasks, suggesting they function better as collaborative tools than autonomous scientific agents.

AtomWorld addresses a critical gap in AI benchmarking by focusing on spatial reasoning and structure manipulation rather than knowledge retrieval alone. Materials science requires constructing and modifying atomic structures—a creative, complex task that has resisted automation. This benchmark evaluates ten fundamental actions across four modeling categories, providing measurable metrics for performance assessment. The findings are sobering: while Claude Opus 4.6 leads the field, performance degrades sharply as task complexity increases, particularly for operations involving spatial relations. Success rates plummet below 12% for rotation tasks, indicating fundamental limitations in how current LLMs conceptualize three-dimensional atomic arrangements. This matters because it reframes expectations around AI's role in scientific research. Rather than replacing human researchers, contemporary LLMs appear positioned as collaborative copilots that accelerate iterative workflows but cannot operate independently on complex structural problems. The benchmark's significance extends beyond evaluation—it serves as a testbed for developing next-generation structure-aware models incorporating reinforcement learning and agentic approaches. These findings suggest the field must invest in architectures specifically designed for spatial reasoning rather than relying on general-purpose language models. For materials science researchers and AI developers, AtomWorld establishes baseline metrics against which improvements can be measured, while highlighting the specific cognitive weaknesses limiting autonomous scientific capability today.

• →Claude Opus 4.6 outperforms other LLMs on atomic structure tasks, but all models show steep performance degradation with increasing complexity.
• →Current LLMs achieve success rates below 12% on rotation operations involving spatial relations, revealing fundamental spatial reasoning deficits.
• →AtomWorld benchmark enables verifiable evaluation metrics across four modeling categories with ten fundamental actions for materials science.
• →LLMs function better as collaborative copilots for structure modeling than as fully autonomous scientific agents in current implementations.
• →The benchmark serves as a development platform for future structure-aware models using reinforcement learning and agentic approaches.