Robust Training of Neural Networks at Arbitrary Precision and Sparsity

Researchers have developed a new framework for training neural networks at ultra-low precision and high sparsity by modeling quantization as additive noise rather than using traditional Straight-Through Estimators. The method enables stable training of A1W1 and sub-1-bit networks, achieving state-of-the-art results for highly efficient neural networks including modern LLMs.

• →The key issue with quantization in neural networks is the absence of proper gradient paths for learning robustness to quantization noise, not just the lack of smoothness.
• →Standard Straight-Through Estimators create instability through mismatched forward and backward passes that don't account for quantization effects.
• →The new framework models quantization as additive noise with a denoising dequantization transform based on ridge regression objectives.
• →The unified approach extends to sparsification by treating it as a special form of quantization that zeros out small values.
• →The method achieves state-of-the-art results for ultra-efficient neural networks and maps efficiency frontiers for modern large language models.