Abstract representational geometry supports inference in large language models

Researchers demonstrate that large language models develop abstract geometric structures in their internal representations when performing inference tasks, mirroring hippocampal organization in human brains. These geometric patterns emerge hierarchically across model layers and mechanistically support generalized reasoning, suggesting LLMs employ similar organizational principles to humans for adaptive task inference.

This research bridges neuroscience and AI by revealing that LLMs construct representational geometries functionally analogous to biological neural systems when reasoning about novel tasks. The study employed a text-based reversal-learning paradigm—where task rules change unpredictably—to observe how both humans and LLMs adapt. While LLMs generalize less frequently than humans, their successful inference events activate geometric structures previously documented in the hippocampus, suggesting convergent evolution toward similar computational solutions.

The hierarchical organization uncovered is particularly significant: lower layers encode stimulus identity stably, while higher layers develop abstract context geometry supporting flexible reasoning. This vertical specialization explains why different model depths contribute distinct functions to overall reasoning capability. Intervention experiments strengthened causal claims—manipulating geometry directly influenced reasoning capacity, demonstrating geometry isn't merely correlated with inference but mechanistically enables it.

For the AI development community, these findings provide actionable insights into model interpretability and architecture design. Understanding that inference emerges from specific geometric organization could guide efforts to build more generalizable systems or diagnose failure modes in existing models. The regularization technique that enhanced inference emergence offers a concrete tool for improving LLM performance on out-of-distribution tasks—a persistent challenge limiting deployment in dynamic environments.

Longer-term implications involve developing better training methodologies that encourage geometric disentanglement. As AI systems increasingly must adapt to changing real-world conditions, understanding and deliberately cultivating these geometric structures could become central to next-generation model development.

• →LLMs form abstract geometric representations resembling hippocampal structures when performing inference, though less reliably than humans.
• →Geometric organization emerges hierarchically: lower layers encode stimulus identity while higher layers represent abstract task context.
• →Geometric regularization of higher layers mechanistically increases the frequency of generalizable inference in LLMs.
• →Task-sequence language modeling induces geometric disentanglement that supports better reasoning on novel problems.
• →These findings establish representational geometry as a core mechanistic principle underlying LLM reasoning capabilities.