Fitting Is Not Enough: Smoothness in Extremely Quantized LLMs

Researchers demonstrate that extreme quantization of large language models causes degradation beyond numerical precision loss, specifically through reduced smoothness in prediction spaces. They introduce smoothness-preserving techniques in post-training and quantization-aware training that improve generation quality independent of numerical accuracy gains.

The paper addresses a critical challenge in making LLMs deployable at scale: extreme quantization reduces model size and computational requirements but introduces subtle degradation mechanisms that existing approaches overlook. While researchers have traditionally focused on maintaining numerical accuracy during quantization, this work reveals that model smoothness—the consistency of predictions across similar inputs—deteriorates independently of precision metrics. As quantization bit-width decreases, the model's prediction neighborhood becomes sparser, constraining the effective token candidates the model can generate and degrading output quality.

This research builds on growing recognition that model compression involves trade-offs beyond raw arithmetic precision. The smoothness degradation effect has implications for real-world deployment scenarios where edge devices and cost-sensitive infrastructure require sub-4-bit models. Current quantization methods insufficient address this phenomenon, creating a gap between theoretical performance metrics and practical generation quality.

For the AI infrastructure and edge computing sectors, this finding guides development of next-generation quantization techniques. Companies deploying LLMs on constrained hardware—mobile devices, IoT systems, or cost-optimized cloud instances—stand to benefit from smoothness-aware quantization methods. The proposed principle integrates into existing quantization frameworks without requiring architectural changes, making adoption straightforward for practitioners.

Future quantization research should incorporate smoothness preservation as a fundamental design criterion alongside accuracy metrics. The work establishes a methodological foundation for understanding quantization effects beyond statistical measures, potentially enabling more sophisticated compression strategies that maintain model behavior quality at extreme compression levels.

• →Extreme quantization degrades model smoothness independently of numerical precision loss, creating sparse prediction neighborhoods.
• →Smoothness-preserving principles improve generation quality in both post-training quantization and quantization-aware training.
• →Degradation severity increases as bit-width decreases, requiring careful consideration for sub-4-bit quantization scenarios.
• →Current quantization algorithms fail to address systematic smoothness degradation despite achieving numerical accuracy targets.
• →Smoothness preservation should become a primary design consideration in future extreme quantization method development.