The Malignant Tail: Spectral Segregation of Label Noise in Over-Parameterized Networks

Researchers identify the 'Malignant Tail' phenomenon where over-parameterized neural networks segregate signal from noise during training, leading to harmful overfitting. They demonstrate that Stochastic Gradient Descent pushes label noise into high-frequency orthogonal subspaces while preserving semantic features in low-rank subspaces, and propose Explicit Spectral Truncation as a post-hoc solution to recover optimal generalization.

• →Over-parameterized networks experience a phase transition from benign to harmful overfitting as noise-to-signal ratio increases.
• →SGD geometrically segregates signal and noise rather than suppressing noise, creating distinct high-frequency and low-rank subspaces.
• →Explicit Spectral Truncation can surgically remove noise-dominated subspaces to restore generalization performance.
• →Excess spectral capacity in neural networks acts as a structural liability that enables noise memorization.
• →This geometric approach provides more stable noise mitigation compared to early stopping techniques.