EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation

EdgeRazor introduces a lightweight quantization framework that compresses large language models to 1.88-bit precision while maintaining performance superior to existing 3-bit methods. The approach combines mixed-precision quantization with knowledge distillation and achieves up to 15.1× faster decoding with 80% storage reduction, requiring significantly lower computational training budgets than comparable techniques.

EdgeRazor addresses a critical bottleneck in AI infrastructure: deploying large language models on resource-constrained devices without sacrificing performance. The framework tackles fundamental tradeoffs in model compression by combining three complementary techniques—mixed-precision quantization, adaptive feature distillation, and entropy-aware KL divergence—to achieve extreme compression ratios that previously required unacceptable accuracy losses.

The quantization landscape has evolved through three main approaches, each with limitations. Post-Training Quantization avoids retraining costs but degrades severely below 4-bit precision. Quantization-Aware Training achieves lower precision but demands substantial computational resources. Existing distillation methods require manual feature selection and teacher-specific datasets. EdgeRazor's innovation lies in automating feature selection through adaptive mechanisms and leveraging teacher output entropy to balance learning objectives dynamically, eliminating manual hyperparameter tuning.

The implications ripple across multiple sectors. Mobile and edge deployment becomes economically viable for sophisticated models—a Qwen3-0.6B model shrinking from 1.41GB to 0.28GB opens deployment possibilities for smartphones, IoT devices, and bandwidth-constrained environments. The 15.1× decoding speedup translates directly to reduced latency and energy consumption. For cloud providers and enterprises, this represents significant cost reduction in inference infrastructure.

The framework's demonstrated superiority over existing 2-bit PTQ methods by 11.3 points while maintaining 4-10× lower training costs suggests EdgeRazor could become an industry standard for production model compression. Future development should examine applicability to newer architectures, mixed-modality models at extreme compression ratios, and real-world deployment across diverse hardware platforms.

• →EdgeRazor achieves 1.88-bit quantization outperforming all existing 3-bit methods with 4-10× lower training costs than leading QAT approaches
• →Storage compression reaches 80% (1.41GB to 0.28GB for Qwen3-0.6B) while accelerating inference by 15.1× relative to 16-bit baselines
• →Entropy-aware KL divergence automatically balances training objectives based on teacher output distribution, eliminating manual hyperparameter selection
• →Framework successfully generalizes across base models, instruction-tuned variants, and multimodal LLMs with consistent performance gains
• →Extreme compression enables practical deployment of sophisticated LLMs on mobile devices and edge hardware with minimal performance degradation