Bounded Hyperbolic Tangent: A Stable and Efficient Alternative to Pre-Layer Normalization in Large Language Models

Researchers propose Bounded Hyperbolic Tanh (BHyT), a normalization technique that replaces Pre-Layer Normalization in large language models, achieving 1.6% faster training and 1.77% higher throughput while maintaining training stability. BHyT addresses the computational overhead and depth-induced instability of current normalization methods by combining tanh with data-driven input bounding and efficient statistics computation.

The research addresses a fundamental challenge in scaling large language models: the tension between computational efficiency and training stability. Pre-Layer Normalization has become the standard approach for LLM training because it provides crucial stability during pretraining and enables effective transfer learning. However, this stability comes at a computational cost, and the method remains vulnerable to performance degradation as model depth increases—a phenomenon where hidden-state magnitudes and variances grow uncontrollably across layers.

The proposed BHyT method emerges from recent trends in efficiency-oriented research that seeks to eliminate or reduce the overhead of normalization layers. While previous normalization-free approaches like Dynamic Tanh improved throughput, they sacrificed stability at scale. BHyT represents a middle ground: it maintains theoretical stability guarantees while recovering efficiency gains by computing exact statistics only once per block and approximating variance in the second normalization pass.

For the AI development community, BHyT's improvements matter because training efficiency directly translates to reduced computational costs and faster model iteration cycles. The 1.6% training speedup compounds significantly across large-scale pretraining runs, potentially lowering barriers to entry for organizations developing LLMs. The maintained performance across language understanding and reasoning benchmarks suggests the approach doesn't sacrifice model quality for speed.

Looking ahead, the key question is adoption rate among major AI labs and whether the theoretical stability guarantees hold across even larger model scales and training runs. The open-source release enables broader testing and potential integration into popular training frameworks, making this a practical contribution rather than purely theoretical.

• →BHyT replaces Pre-Layer Normalization with a tanh-based method that improves training speed by 1.6% and token generation throughput by 1.77% while maintaining stability.
• →The method prevents activation magnitude and variance growth across deep layers through data-driven input bounding and explicit theoretical stability guarantees.
• →BHyT computes full statistics once per block and uses lightweight variance approximation for the second normalization, reducing computational overhead.
• →Empirical results show maintained performance on language understanding and reasoning benchmarks, indicating no quality trade-off for efficiency gains.
• →Open-source code availability accelerates potential adoption and integration into mainstream LLM training frameworks.