On the Rate of Convergence of GD in Non-linear Neural Networks: An Adversarial Robustness Perspective

Researchers prove that gradient descent in neural networks converges to optimal robustness margins at an extremely slow rate of Θ(1/ln(t)), even in simplified two-neuron settings. This establishes the first explicit lower bound on convergence rates for robustness margins in non-linear models, revealing fundamental limitations in neural network training efficiency.

• →Gradient descent converges to optimal robustness margins but at a prohibitively slow rate of Θ(1/ln(t)) even in minimal neural network settings.
• →This is the first explicit lower bound established for convergence rates of robustness margins in non-linear models.
• →The slow convergence pattern is pervasive across multiple natural network initializations, suggesting a fundamental limitation.
• →Researchers developed rigorous mathematical analysis to control gradient descent trajectories in non-linear architectures.
• →The study reveals inherent efficiency challenges in neural network training for adversarial robustness.