How Pruning Reshapes Features: Sparse Autoencoder Analysis of Weight-Pruned Language Models

Researchers conducted the first systematic study of how weight pruning affects language model representations using Sparse Autoencoders across multiple models and pruning methods. The study reveals that rare features survive pruning better than common ones, suggesting pruning acts as implicit feature selection that preserves specialized capabilities while removing generic features.

• →Rare SAE features with low firing rates survive pruning significantly better than frequent features across most experimental conditions.
• →Wanda pruning preserves feature structure up to 3.7x better than magnitude pruning methods.
• →Pre-trained SAEs remain viable on Wanda-pruned models up to 50% sparsity levels.
• →Pruning acts as implicit feature selection, preferentially destroying high-frequency generic features while preserving specialized rare ones.
• →Geometric feature survival does not predict causal importance, revealing a key dissociation for interpretability research.