Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention

Researchers propose Affine-Scaled Attention, a new mechanism that improves Transformer model training stability by introducing flexible scaling and bias terms to attention weights. The approach shows consistent improvements in optimization behavior and downstream task performance compared to standard softmax attention across multiple language model sizes.

• →Affine-Scaled Attention relaxes strict normalization constraints in Transformer attention while maintaining value representation aggregation.
• →The method introduces input-dependent scaling and bias terms to softmax-normalized attention weights for better control.
• →Experiments demonstrate improved training stability and optimization behavior across multiple large-scale language model sizes.
• →The approach outperforms both standard softmax attention and attention sink baselines on downstream tasks.
• →Results suggest that modest attention reweighting provides a practical way to enhance Transformer model performance.