EPTS: Elastic Post-Training Sparsity for Efficient Large Language Model Compression

Researchers introduce EPTS, a new framework for compressing large language models that enables a single optimized model to perform efficiently across multiple sparsity levels, eliminating the need for separate optimization for each deployment scenario. This approach combines Multi-Sparsity Hierarchy LoRA and a Feature Mixer mechanism to maintain performance while reducing computational requirements.

The advancement of large language model compression addresses a critical bottleneck in AI deployment. Current post-training sparsity methods require individual optimization for each target sparsity level, creating inefficiency when deploying models across diverse hardware configurations ranging from data centers to edge devices. EPTS fundamentally shifts this paradigm by enabling multi-scenario deployment from a single optimization pass, reducing time-to-deployment and computational overhead.

This work builds on the growing trend of model efficiency research, following methodologies like SparseGPT and Wanda that have demonstrated effective parameter pruning. The innovation lies in its multi-sparsity flexibility rather than focusing solely on achieving maximum compression at one level. By introducing MS-HiLoRA to facilitate knowledge inheritance across sparsity levels and MSFM to dynamically adapt to pruning perturbations, the framework addresses fundamental challenges in maintaining model quality during compression.

For the AI industry, this development has practical implications. Organizations deploying LLMs across heterogeneous hardware environments can significantly reduce infrastructure costs and optimization cycles. Developers gain flexibility to adjust sparsity levels post-deployment without retraining, enabling responsive adaptation to changing resource constraints. The competitive performance compared to established methods suggests the approach balances efficiency gains with model quality.

Looking forward, the open-source release on GitHub enables broad adoption and community refinement. Future developments may extend these techniques to multimodal models or explore dynamic sparsity adjustment during inference. The framework's success on LLaMA and OPT families suggests potential applicability to other model architectures, making it a significant contribution to the broader LLM efficiency ecosystem.

• →EPTS enables single-model deployment across multiple sparsity levels, eliminating costly per-sparsity optimization cycles
• →MS-HiLoRA mechanism implements knowledge inheritance to reduce parameter competition during compression
• →Multi-Sparsity Feature Mixer dynamically adapts model representations to varying pruning perturbations
• →Experimental results match state-of-the-art methods while offering superior deployment flexibility
• →Open-source availability accelerates adoption across LLM deployment scenarios