Preserving Plasticity in Continual Learning via Dynamical Isometry

Researchers identify dynamical isometry—maintaining consistent layer-wise Jacobian singular values—as a mechanism for preserving neural network plasticity during continual learning under non-stationary conditions. They propose AdamO, an adaptive optimizer combining isometry regularization with gradient updates, demonstrating improved performance across supervised and reinforcement-learning benchmarks where traditional networks suffer progressive learning degradation.

The research addresses a fundamental challenge in deep learning: neural networks lose their ability to learn (plasticity) when trained continuously on changing data distributions. This phenomenon limits practical applications in dynamic environments, from robotics to adaptive AI systems. The authors establish a mathematical connection between plasticity loss and the Neural Tangent Kernel, proposing dynamical isometry—a property where layer-wise transformations maintain proportional signal magnitudes—as a stabilizing mechanism.

The work builds on decades of neural network research exploring gradient flow and representational capacity. Prior approaches attempted to preserve plasticity through various heuristics, but lacked unified theoretical grounding. This paper reinterprets existing methods through the isometry lens, revealing they address only partial aspects of the problem. The identification of dynamical isometry as a unifying principle represents conceptual progress in understanding continual learning dynamics.

For AI practitioners, particularly those developing systems requiring continual adaptation, the AdamO optimizer offers a practical tool with theoretical justification. The regularization scheme's efficiency matters for large-scale deployments. The discovery that isometry regularization can reactivate dormant ReLU units suggests computational benefits beyond plasticity preservation.

The research implications extend to reinforcement learning agents operating in non-stationary environments and federated learning systems encountering diverse data streams. Future work likely focuses on extending these principles to modern architectures like transformers and exploring computational trade-offs at scale. The theoretical framework may influence optimizer design across the field.

• →Dynamical isometry preserves neural network plasticity during continual learning on non-stationary data
• →AdamO optimizer decouples isometry regularization from gradient updates, improving performance on benchmarks
• →Prior plasticity-preserving methods only address partial measures of isometry, missing key mechanisms
• →Isometry regularization can reactivate dormant ReLU units, providing computational efficiency benefits
• →Framework applies across supervised learning and reinforcement learning continual-learning scenarios