#convergence-analysis News & Analysis

3 articles tagged with #convergence-analysis. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

3 articles

AIBullisharXiv – CS AI · Mar 37/104

🧠

A Convergence Analysis of Adaptive Optimizers under Floating-point Quantization

Researchers introduce the first theoretical framework analyzing convergence of adaptive optimizers like Adam and Muon under floating-point quantization in low-precision training. The study shows these algorithms maintain near full-precision performance when mantissa length scales logarithmically with iterations, with Muon proving more robust than Adam to quantization errors.

AIBullisharXiv – CS AI · 6d ago6/10

🧠

New Hybrid Fine-Tuning Paradigm for LLMs: Algorithm Design and Convergence Analysis Framework

Researchers propose a novel hybrid fine-tuning method for Large Language Models that combines full parameter updates with Parameter-Efficient Fine-Tuning (PEFT) modules using zeroth-order and first-order optimization. The approach addresses computational constraints of full fine-tuning while overcoming PEFT's limitations in knowledge acquisition, backed by theoretical convergence analysis and empirical validation across multiple tasks.

AINeutralarXiv – CS AI · Mar 45/103

🧠

Why Adam Can Beat SGD: Second-Moment Normalization Yields Sharper Tails

Research paper establishes the first theoretical separation between Adam and SGD optimization algorithms, proving Adam achieves better high-probability convergence guarantees. The study provides mathematical backing for Adam's superior empirical performance through second-moment normalization analysis.