On the Geometric Structure of Layer Updates in Deep Language Models

Researchers analyzed the geometric structure of layer updates in deep language models, finding they decompose into a dominant tokenwise component and a geometrically distinct residual. The study shows that while most updates behave like structured reparameterizations, functionally significant computation occurs in the residual component.

• →Layer updates in deep language models can be decomposed into dominant tokenwise components and geometrically distinct residuals.
• →The full layer update aligns almost perfectly with the tokenwise component across multiple architectures including Transformers.
• →The residual component exhibits weaker alignment and larger angular deviation, indicating it's not just a small correction.
• →Approximation error under restricted tokenwise models strongly correlates with output perturbation, with correlations up to 0.95.
• →The framework provides an architecture-agnostic method for probing geometric and functional structure in modern language models.