Grokking as Dimensional Phase Transition in Neural Networks

Researchers identify neural network 'grokking' as a dimensional phase transition where effective dimensionality shifts from sub-diffusive to super-diffusive during the memorization-to-generalization transition. The study reveals this transition reflects gradient field geometry rather than network architecture, offering new insights into overparameterized network trainability.

• →Grokking represents a dimensional phase transition with effective dimensionality crossing from D < 1 to D > 1 at generalization onset.
• →The transition exhibits self-organized criticality and is driven by gradient field geometry, not network architecture.
• →Synthetic Gaussian gradients maintain D ≈ 1 regardless of topology, while real training shows dimensional excess from backpropagation correlations.
• →The dimensional crossing is robust across different network topologies and scales.
• →This discovery provides new theoretical framework for understanding trainability in overparameterized networks.