A Convergence Analysis of Adaptive Optimizers under Floating-point Quantization

Researchers introduce the first theoretical framework analyzing convergence of adaptive optimizers like Adam and Muon under floating-point quantization in low-precision training. The study shows these algorithms maintain near full-precision performance when mantissa length scales logarithmically with iterations, with Muon proving more robust than Adam to quantization errors.

• →First theoretical framework developed for analyzing adaptive optimizer convergence under hardware-aware floating-point quantization
• →Both Adam and Muon maintain convergence rates close to full-precision versions with proper quantization parameters
• →Adam shows high sensitivity to weights and second-moment quantization due to its β₂ → 1 parameter dependency
• →Muon optimizer demonstrates superior robustness to quantization errors compared to Adam
• →Mantissa length needs to scale only logarithmically with iteration count to preserve performance