#encoder-design News & Analysis

Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders

Researchers propose a modified Transformer encoder that explicitly separates positional and semantic information into three independent streams, revealing that positional data naturally collapses into a low-frequency 2D structure and that standard encoding methods fail to preserve macroscopic positional information under language modeling pressure.