Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models

Researchers demonstrate that scale vectors in large language models, despite comprising negligible model parameters, significantly impact training performance and optimization. Through theoretical analysis and empirical validation across models from 0.12B to 2B parameters, the study proposes three complementary improvements to scale vector design that enhance training efficiency without adding computational overhead.

Scale vectors represent a fascinating intersection of theoretical understanding and practical model optimization in modern LLMs. While normalization layers have received substantial research attention, the learned scale components have remained largely overlooked despite their consistent presence across architectures. This research bridges that gap by systematically examining why these tiny parameter sets produce outsized effects on model training dynamics.

The findings reveal that scale vectors function primarily as optimization mechanisms rather than expressivity enhancers in Pre-Norm architectures. Through preconditioning effects on subsequent linear mappings, they create a self-amplifying mechanism that improves gradient flow and convergence properties. The distinction between Input-Norm and Output-Norm layers proves critical, with weight decay regularization showing opposing benefits depending on layer type—a nuance that typical hyperparameter tuning might miss.

For the AI development community, these insights carry practical implications for model architecture design and training optimization. The proposed improvements—branch-specific heterogeneity, strategic placement near linear mappings, and magnitude-direction reparameterization—represent low-cost enhancements that consistently lower terminal loss and improve scaling behavior across different model sizes and optimizers. The research demonstrates these gains hold under industrial-scale token budgets, suggesting real applicability in production environments.

Looking forward, this work invites deeper investigation into other 'negligible' components within LLMs that may similarly exert disproportionate influence on model behavior. As scaling laws and efficiency become increasingly important in AI development, understanding these subtle optimization mechanisms could drive meaningful improvements in model training efficiency and resource utilization across the field.

• →Scale vectors in LLMs significantly improve training despite comprising negligible model parameters through preconditioning effects
• →Weight decay affects Input-Norm and Output-Norm layers oppositely, suggesting layer-specific regularization strategies are beneficial
• →Proposed improvements to scale vector design consistently reduce terminal loss across 0.12B-2B parameter models with minimal overhead
• →Scale vectors enhance optimization but not expressivity in Pre-Norm architectures, clarifying their fundamental role in neural networks
• →Research validates improvements across multiple optimizers and learning rate schedules under industrial-scale training budgets