Emergent Ordinal Geometry in Transformers Trained on Local Comparisons

Researchers demonstrate that Transformers trained exclusively on adjacent comparisons spontaneously develop one-dimensional geometric structures that encode hidden rank orderings, exhibiting the symbolic distance effect observed in animal cognition. This discovery mechanistically bridges cognitive science with neural network representations, showing that decision confidence scales with ordinal distance even at ceiling accuracy.

This research addresses a fundamental question about how neural networks learn abstract relational reasoning without explicit instruction. By training Transformers on transitive inference—inferring A < C from A < B and B < C—researchers uncovered an emergent geometric phenomenon where entity embeddings collapse onto a one-dimensional manifold that perfectly recovers the hidden ordering. The finding is significant because it explains how networks solve out-of-distribution generalization through learned geometry rather than symbolic logic chains.

The work connects decades of behavioral neuroscience showing humans and animals use mental number lines for comparisons, not logical reasoning. The symbolic distance effect—the empirical observation that distant comparisons are faster and more accurate than nearby ones—appears in the network's decision confidence and geometric separation, despite achieving perfect accuracy on all comparisons. This convergence between biological cognition and artificial neural networks suggests deep principles about how intelligent systems represent ordinal information.

For the AI research community, these results illuminate how inductive biases and optimization dynamics shape learned representations. The emergence of grokking-like transient behavior during training indicates that geometric reorganization doesn't happen smoothly but exhibits phase transitions. This mechanistic understanding could inform architecture design for relational reasoning tasks. The research also validates approaches combining cognitive science with deep learning interpretability, potentially accelerating progress in explainable AI by grounding high-dimensional computations in interpretable geometric structures.

• →Transformers spontaneously learn one-dimensional ordinal structures from adjacent comparisons without explicit geometric supervision.
• →Emergent embeddings exhibit the symbolic distance effect, directly mirroring 50 years of behavioral neuroscience findings across species.
• →Out-of-distribution generalization correlates with geometric reorganization and grokking-like dynamics during training.
• →Decision confidence scales monotonically with ordinal distance even when accuracy reaches ceiling performance.
• →Results suggest neural networks and biological cognition share fundamental principles for representing relational structure.