Unifying Learning Dynamics and Generalization in Transformers Scaling Law

Researchers formalize the theoretical foundations of LLM scaling laws by modeling transformer learning dynamics as differential equations, establishing matching upper and lower bounds that characterize a two-phase convergence pattern: exponential decay during optimization followed by power-law decay during the statistical phase. This work bridges the gap between empirical observations and rigorous mathematical theory, providing independent scaling relationships for model size, training time, and dataset size.

The scaling law has become fundamental to LLM development, yet its theoretical justification has remained largely empirical. This research addresses that gap by formalizing transformer learning dynamics through rigorous mathematical analysis rather than toy models, examining how computational resources translate into performance improvements across real-world conditions.

The key innovation lies in characterizing a two-phase convergence structure. Initially, excess risk decays exponentially with computational cost, representing pure optimization efficiency. Beyond a critical resource threshold, the system transitions to a statistical phase where generalization error follows a $\Theta(C^{-1/7})$ power-law decay. This phase transition explains why additional compute shows diminishing returns—a phenomenon practitioners have observed but lacked theoretical justification for. The matching upper and lower bounds (certified through information-theoretic and oracle arguments) make these rates tight, providing confidence in the theoretical predictions.

For the AI industry, this work validates design decisions around model scaling and resource allocation. Teams can now make data-informed decisions about when to scale model size versus increase training compute or dataset size, each governed by distinct scaling relationships. The theoretical framework enables researchers to predict scaling behavior more accurately without exhaustive empirical testing.

Looking ahead, this mathematical foundation could accelerate AI development by reducing expensive exploration of the scaling frontier. Subsequent work may refine the condition-number gap and logarithmic factors, potentially tightening the bounds further. Understanding these phase transitions could also inform architecture innovations that shift the statistical phase threshold, fundamentally improving training efficiency.

• →Scaling laws exhibit a two-phase structure with exponential optimization decay followed by power-law statistical phase decay at $\Theta(C^{-1/7})$
• →Mathematical bounds on excess risk are now proven tight up to constants and logarithmic factors, validating empirical scaling observations
• →Model size, training time, and dataset size each follow independent scaling relationships that can guide resource allocation decisions
• →The critical resource threshold marks the transition from optimization-limited to data/generalization-limited regimes
• →Rigorous theoretical framework reduces reliance on expensive empirical scaling studies for predicting LLM performance