Residual Connections and the Causal Shift: Uncovering a Structural Misalignment in Transformers

Researchers identified a structural misalignment in Transformer models where residual connections tie to current tokens while supervision targets next tokens. They propose lightweight residual attenuation techniques that improve autoregressive Transformer performance by addressing this input-output alignment shift.

• →Large Language Models have a subtle misalignment between residual connections and next-token prediction targets.
• →Hidden token representations switch from input to output alignment deep within the network architecture.
• →Researchers propose residual attenuation as a lightweight solution to address this structural issue.
• →The proposed mitigation can be implemented as fixed-layer intervention or learnable gating mechanism.
• →Experiments show the approach alleviates representation misalignment and improves benchmark performance.