Scaling Laws for Precision in High-Dimensional Linear Regression

Researchers developed theoretical scaling laws for low-precision AI model training, analyzing how quantization affects model performance in high-dimensional linear regression. The study reveals that multiplicative and additive quantization schemes have distinct effects on effective model size, with multiplicative maintaining full precision while additive reduces it.

• →Low-precision training optimization requires joint allocation of model size, dataset size, and numerical precision to balance quality and costs.
• →Multiplicative quantization maintains full-precision model size while additive quantization reduces effective model size.
• →Both quantization schemes introduce additive error and degrade effective data size but with different scaling behaviors.
• →The research provides theoretical foundation for optimizing AI training protocols under hardware constraints.
• →Numerical experiments validated the theoretical findings on quantization's impact on model training efficiency.