#edge-of-stability News & Analysis

3 articles tagged with #edge-of-stability. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

3 articles

AINeutralarXiv – CS AI · Jun 56/10

🧠

Gradient Descent with Large Step Size Restores Symmetry in Deep Linear Networks with Multi-Pathway

Researchers demonstrate that discrete Gradient Descent with large step sizes produces fundamentally different training dynamics in deep linear networks compared to continuous Gradient Flow. Their analysis reveals that multi-pathway networks redistribute signals across pathways during later training stages rather than concentrating them in single pathways, challenging prevailing theoretical predictions and suggesting that optimization step size significantly influences neural network representation learning.

AINeutralarXiv – CS AI · Jun 56/10

🧠

Gradient descent at the Edge of Stability: free energy model and kinetic description of the two-layer network

Researchers propose a continuous-time mathematical model for analyzing gradient descent dynamics in the Edge of Stability regime, where large learning rates cause oscillations in neural network training. The model introduces an effective free energy framework that combines risk with a curvature-related term, enabling better prediction of training dynamics in wide two-layer networks and validated on matrix factorization and CIFAR-10 tasks.

AINeutralarXiv – CS AI · May 116/10

🧠

A Rod Flow Model for Adam at the Edge of Stability

Researchers extend rod flow modeling to Adam and other adaptive gradient methods, enabling more accurate continuous-time analysis of optimizer behavior at the edge of stability. This advancement bridges a gap in theoretical understanding of momentum-based optimization algorithms critical to modern deep learning.