Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning

Researchers identify that deep neural networks lose plasticity during continual learning due to Hessian spectral collapse, where curvature information vanishes and prevents gradient-based optimization. The study proposes regularization techniques combining high effective feature rank maintenance and L2 penalties to preserve learning capacity across sequential tasks.

Deep neural networks face a fundamental challenge in continual learning: they progressively lose the ability to acquire new information without catastrophic forgetting or performance degradation. This research identifies spectral collapse—a mathematical phenomenon where the Hessian matrix loses meaningful curvature directions—as the underlying cause of plasticity loss. The finding bridges theoretical understanding and practical optimization by establishing that the loss-weighted Gram matrix exhibits spectral equivalence to the Generalized Gauss-Newton approximation, connecting Neural Tangent Kernel dynamics to curvature properties. This theoretical insight enables more targeted interventions than previous approaches.

The practical implications extend across machine learning applications requiring sequential task learning, from robotics to natural language processing. Networks that maintain high plasticity can adapt to distribution shifts and new objectives without expensive retraining from scratch. The proposed dual regularization strategy—preserving effective feature rank while applying L2 penalties—demonstrates empirical validation on both supervised and reinforcement learning benchmarks. This approach directly targets the spectral collapse phenomenon rather than applying generic regularization.

For practitioners developing production systems, this research provides actionable mechanisms to improve continual learning robustness. The Kronecker factored Hessian approximation offers computational efficiency compared to full second-order methods, making practical deployment feasible. As AI systems increasingly face real-world environments with non-stationary data distributions, maintaining plasticity becomes essential for long-term performance. The work enables more principled design of neural architectures and training procedures that balance stability and adaptability, advancing the field's capability to deploy genuinely adaptive AI systems.

• →Hessian spectral collapse causes plasticity loss in continual learning by eliminating meaningful curvature directions for optimization.
• →Maintaining high effective feature rank and applying L2 regularization together effectively preserve network plasticity across sequential tasks.
• →Theoretical analysis proves loss-weighted Gram matrices exhibit spectral equivalence to Generalized Gauss-Newton approximations.
• →The approach improves both supervised and reinforcement learning performance on continual learning benchmarks.
• →Kronecker factored Hessian approximations provide computationally efficient implementations for practical deployment.