3 articles tagged with #low-precision-training. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers demonstrate that nGPT, a neural architecture that normalizes weights and hidden representations to a unit hypersphere, achieves stable 4-bit precision training without requiring additional quantization interventions. The approach leverages mathematical properties of dot products to maintain stronger signal-to-noise ratios, enabling efficient training of models up to 30B parameters.
Researchers introduce the first theoretical framework analyzing convergence of adaptive optimizers like Adam and Muon under floating-point quantization in low-precision training. The study shows these algorithms maintain near full-precision performance when mantissa length scales logarithmically with iterations, with Muon proving more robust than Adam to quantization errors.
Researchers establish information-theoretic lower bounds for bit-constrained stochastic optimization, proving that B-bit quantized gradients require communication overhead of TB = Omega(d) and statistical complexity of T = Omega(sigma^2 d / eps^2 * max{1, d/B}). The work provides the first rigorous characterization of what's theoretically possible in low-precision pretraining, contrasting with existing empirical studies of FP8 and MXFP4 systems.
