#optimization-dynamics News & Analysis

2 articles tagged with #optimization-dynamics. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

2 articles

AIBullisharXiv – CS AI · May 297/10

🧠

Overcoming Forgetting in LLM Fine-Tuning with Evolution Strategies

Researchers demonstrate that Evolution Strategies (ES) can effectively fine-tune large language models without catastrophic forgetting of prior tasks, contrary to recent concerns. By introducing Anchored Weight Decay (AWD), a regularization technique that constrains optimization toward initial parameters, the work shows ES-based continual learning is viable and computationally efficient compared to reinforcement learning approaches.

AINeutralarXiv – CS AI · May 116/10

🧠

Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer

Researchers develop a dynamical mean-field theory framework to analyze how neural network weight spectra evolve during training, revealing that different parameterization schemes (μP vs NTK) produce fundamentally different outlier dynamics. The findings suggest that neural scaling laws and hyperparameter transfer depend critically on how outlier eigenvalues behave, with implications for understanding deep learning generalization and optimization.