Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders

Researchers propose a modified Transformer encoder that explicitly separates positional and semantic information into three independent streams, revealing that positional data naturally collapses into a low-frequency 2D structure and that standard encoding methods fail to preserve macroscopic positional information under language modeling pressure.

This research addresses a fundamental gap in Transformer architecture understanding by mechanistically studying how positional encoding functions within neural networks. The authors demonstrate that positional and semantic signals naturally occupy nearly orthogonal subspaces, enabling them to create a disentangled architecture that processes these streams independently. This architectural modification provides unprecedented insight into internal mechanisms that have remained opaque in standard Transformers.

The findings reveal critical limitations in current positional encoding methods like RoPE, which struggle with long-context understanding and retrieval tasks. By isolating positional information, the researchers discovered that absolute positional (AP) representations spontaneously organize into a low-frequency 2D manifold reflecting document structure, while relative positional (RP) information exclusively supports semantic-oriented attention. Crucially, standard methods fail to robustly retain this macroscopic structure under masked language modeling pressure, with positional encoding information degrading in final layers.

These mechanistic insights carry practical implications for improving large language models, particularly for long-context applications where positional encoding currently constrains performance. The disentangled approach improved linguistic representation performance on 49 of 65 linguistic phenomena tested, suggesting measurable benefits in downstream tasks. The work establishes a new framework for understanding and potentially designing superior positional encoding methods that could enhance Transformer capabilities for retrieval-augmented generation, document understanding, and extended context windows.

Future research should explore whether these disentangled mechanisms translate to improved performance on demanding real-world tasks and whether the insights inform next-generation architecture designs beyond standard Transformers.

• →Positional and semantic information occupy nearly orthogonal subspaces in Transformers, enabling explicit architectural disentanglement.
• →Absolute positional representations spontaneously collapse into low-frequency 2D manifolds that encode document structure.
• →Standard positional encodings including RoPE fail to robustly preserve macroscopic structure under language modeling training.
• →Disentangled positional encoding improves linguistic representation performance on 75% of tested linguistic phenomena.
• →Attention heads naturally specialize into structure-oriented and semantic-oriented groups with distinct positional roles.