#second-order-methods News & Analysis

2 articles tagged with #second-order-methods. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

2 articles

AINeutralarXiv – CS AI · Jun 46/10

🧠

Why Muon Outperforms Adam: A Curvature Perspective

Researchers demonstrate that Muon, an optimizer for large language model training, outperforms Adam by approximately 2x efficiency through lower Normalized Directional Sharpness (NDS) rather than smaller update scales. Using curvature analysis and stylized quadratic problems, the work reveals that Muon's advantage stems from better balancing of update energy across heterogeneous curvature regions, with benefits amplified in data-imbalanced scenarios.

AINeutralarXiv – CS AI · Jun 26/10

🧠

FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoo

Researchers propose FOAM, an adaptive algorithm that addresses the computational bottleneck in Shampoo optimization by dynamically controlling damping factors and eigendecomposition frequency to mitigate errors from stale preconditioner updates. The method reduces wall-clock training time while maintaining convergence stability, offering a practical solution to the efficiency-fidelity trade-off in large-scale machine learning optimization.