GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models

Researchers introduce GPrune-LLM, a new structured pruning framework that improves compression of large language models by addressing calibration bias and cross-task generalization issues. The method partitions neurons into behavior-consistent modules and uses adaptive metrics based on distribution sensitivity, showing consistent improvements in post-compression performance.

• →Current LLM pruning methods suffer from calibration bias when estimating neuron importance from single datasets.
• →GPrune-LLM identifies that neurons exhibit heterogeneous distribution sensitivity with varying cross-dataset ranking consistency.
• →The framework partitions neurons into behavior-consistent modules to localize ranking competition and prevent important neurons from being crowded out.
• →For unreliable modules, the method switches from activation-based to activation-independent importance metrics.
• →Experiments demonstrate consistent improvements in post-compression generalization, especially at high sparsity levels.