BitsMoE: Efficient Spectral Energy-Guided Bit Allocation for MoE LLM Quantization

BitsMoE introduces a spectral-energy-guided quantization framework for compressing Mixture-of-Experts large language models, achieving significant improvements in the ultra-low-bit regime. The method uses SVD decomposition to intelligently allocate bits across expert weights, delivering 27.83 percentage point accuracy improvements over existing approaches at 2-bit quantization while accelerating inference speed by 1.76× on Qwen models.

BitsMoE addresses a critical bottleneck in deploying large MoE language models: memory efficiency without catastrophic accuracy loss. While MoE architectures reduce computational overhead through sparse expert activation, they require keeping all expert weights in memory, creating deployment constraints. Existing quantization approaches either irreversibly prune model capacity or fail to account for the heterogeneous importance of different expert components, making them unsuitable for ultra-low-bit scenarios where precision is severely constrained.

The innovation lies in BitsMoE's SVD-based decomposition strategy, which separates shared cross-expert structure from expert-specific components. By preserving the shared basis without quantization and treating expert-specific factors as discrete quantization units, the framework maintains model coherence while enabling fine-grained bit allocation. The activation-aware integer linear programming optimization ensures bit budgets target the most impactful weights, minimizing reconstruction loss in a principled manner.

For the AI infrastructure sector, these results have practical implications. The 1.76× speedup in decoding and 12.3× faster quantization process reduce deployment costs and latency-sensitive applications. The 27.83 percentage point accuracy improvement at 2-bit quantization suggests MoE models can operate effectively in memory-constrained environments—edge devices, mobile inference, and cost-sensitive cloud deployments. This democratizes access to large model capabilities and reduces operational expenses for AI service providers.

The public availability of code and models accelerates adoption across the community. Future developments may extend these techniques to other sparse architectures or combine them with other compression methods. The benchmark improvements demonstrate that careful algorithmic design can overcome fundamental trade-offs between model size and performance.

• →BitsMoE achieves 27.83 percentage point accuracy improvements over GPTQ at 2-bit quantization on Qwen3-30B models
• →SVD decomposition preserves shared expert structure while enabling fine-grained bit allocation across heterogeneous components
• →Decoding speed increases 1.76× with 12.3× faster quantization process compared to existing methods
• →The framework uses activation-aware integer linear programming to optimize bit allocation under fixed memory budgets
• →Open-source implementation enables rapid adoption for memory-constrained MoE LLM deployment scenarios