Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descent

Researchers propose a representation-readout decomposition framework that explains anomalous neural network training phenomena like grokking and double descent by analyzing two competing learning processes: representation learning in encoders and readout calibration in classifiers. The framework provides task-agnostic diagnostics that reveal these phenomena stem from fluctuations in relative learning speeds rather than mysterious delays, challenging existing lazy-to-rich learning theories.

This theoretical contribution addresses a fundamental gap in deep learning interpretability by providing a unifying explanation for two puzzling training dynamics that have troubled practitioners. Grokking—where test accuracy suddenly jumps after extended training despite continued loss reduction—and double descent—where test loss paradoxically rises before falling again—have resisted unified explanation because existing analyses remain task-specific and architecturally dependent.

The representation-readout decomposition framework elegantly sidesteps these limitations by decomposing learning into two parallel processes with measurable dynamics. Rather than attributing grokking to mysterious threshold phenomena, the authors demonstrate that readout overfitting precedes representation maturation, creating apparent delays in generalization. This insight reframes delayed generalization not as anomalous but as a natural consequence of asynchronous learning speeds.

The framework's diagnostic power extends beyond theoretical elegance. By identifying representation degradation and readout misalignment as signatures of spurious generalization, researchers can distinguish genuine learning improvements from artifacts of non-standard training recipes. This capability has immediate practical value for practitioners designing training procedures and validating model performance claims.

For the AI research community, this work strengthens the foundations of neural network theory by replacing task-specific explanations with mechanistic understanding. The use of representational geometry and neural tangent kernel analysis demonstrates that interpretability can emerge from mathematically rigorous decomposition rather than empirical observation alone. These insights will likely influence how researchers design architectures and training protocols, particularly in domains where grokking or double descent behaviors have been observed.

• →Grokking and double descent arise from misaligned learning speeds between representation learning and readout calibration rather than fundamental algorithmic thresholds.
• →Readout overfitting precedes representation maturation in grokking, contradicting simpler lazy-to-rich learning accounts of network training.
• →The framework provides diagnostic signatures to distinguish genuine generalization from artifacts caused by non-standard training procedures.
• →Representation-readout decomposition offers a task-agnostic analysis tool applicable across diverse architectures and domains.
• →Understanding these learning dynamics enables better architectural design and training recipe validation in deep neural networks.