MUXQ: Mixed-to-Uniform Precision MatriX Quantization via Low-Rank Outlier Decomposition

Researchers propose MUXQ, a new quantization technique for large language models that addresses activation outliers through low-rank decomposition. The method enables efficient INT8 quantization while maintaining accuracy close to FP16, making it suitable for edge device deployment with NPU-based hardware.

• →MUXQ introduces an auxiliary matrix to redistribute outlier magnitudes across channels, solving a key limitation in existing quantization methods.
• →The technique enables both activations and weights to be quantized to INT8 precision while preserving accuracy comparable to FP16.
• →Testing on GPT-2 models (0.1B to 0.7B parameters) shows consistently lower perplexity than naive quantization approaches.
• →The method is designed for NPU-based edge devices where integer computation is more efficient than floating-point operations.
• →MUXQ can be combined with other quantization techniques and adds only modest computational overhead.