The Geometry of Grokking: Norm Minimization on the Zero-Loss Manifold

Researchers provide a mathematical framework explaining grokking—the phenomenon where neural networks suddenly generalize after memorizing training data. The study proves that gradient descent minimizes weight norms on the zero-loss manifold and derives closed-form expressions for post-memorization dynamics, offering theoretical clarity on this previously elusive learning behavior.

This research addresses a fundamental mystery in deep learning: why neural networks experience sudden generalization jumps long after perfectly memorizing training data. The authors move beyond previous weight-decay explanations by formulating grokking as constrained optimization, where gradient descent acts to minimize parameter norms while maintaining zero training loss. This geometric interpretation provides rigorous mathematical grounding for what was previously observed but poorly understood.

Grokking has puzzled researchers since its discovery because it defies intuition about machine learning—models appear to waste computation memorizing before suddenly learning general patterns. Prior work attributed this to representation learning driven by regularization, but lacked precise dynamics. This paper bridges that gap through formal proofs in limiting cases and introduces a decoupling approximation that isolates specific parameter learning dynamics, enabling closed-form solutions for simplified architectures.

The implications extend beyond academic curiosity into practical AI development. Understanding grokking mechanics helps practitioners predict when and why delayed generalization occurs, potentially improving training efficiency and model reliability. For the broader AI field, this work demonstrates how geometric and optimization perspectives can illuminate neural network behavior that statistical analysis alone cannot explain. The formal framework may inform future architecture designs and training strategies.

Future research should validate these theoretical predictions across diverse architectures and datasets, examine whether grokking's benefits justify its computational overhead, and explore whether the insights apply to large-scale modern models where grokking-like phenomena may occur undetected.

• →Grokking is formalized as weight norm minimization on the zero-loss manifold, providing rigorous mathematical explanation for delayed generalization
• →Researchers prove the mechanism in the limit of infinitesimal learning rates and derive closed-form dynamics for two-layer networks
• →The geometric framework decouples parameter learning, enabling tractable analysis of previously opaque neural network behavior
• →Experimental validation confirms predictions match observed grokking phenomena including representation learning patterns
• →Understanding grokking dynamics could improve training efficiency and inform architecture design for more predictable generalization