#normalization-layers News & Analysis

2 articles tagged with #normalization-layers. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

2 articles

AIBullisharXiv – CS AI · Jun 57/10

🧠

SpanNorm: Reconciling Training Stability and Performance in Deep Transformers

Researchers introduce SpanNorm, a novel normalization technique for deep Transformer architectures that combines the training stability of PreNorm with the performance benefits of PostNorm. The method uses spanning residual connections and PostNorm-style computation to prevent gradient instability and representation collapse, demonstrating improvements in both dense and Mixture-of-Experts model configurations.

AINeutralarXiv – CS AI · May 276/10

🧠

Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models

Researchers demonstrate that scale vectors in large language models, despite comprising negligible model parameters, significantly impact training performance and optimization. Through theoretical analysis and empirical validation across models from 0.12B to 2B parameters, the study proposes three complementary improvements to scale vector design that enhance training efficiency without adding computational overhead.