Towards Data-free and Training-free Compression for Speech Foundation Models Using Parameter Clustering

Researchers present a novel compression technique for speech foundation models using parameter clustering and k-means pruning without requiring training data or fine-tuning. The method demonstrates significant performance improvements over traditional magnitude-based pruning on HuBERT-large and Whisper-large-v3, with 27-59% relative WER reductions at various sparsity levels.

The paper addresses a critical challenge in deploying large speech foundation models: reducing computational overhead while maintaining performance. Traditional model compression techniques often require access to training data and extensive retraining, creating barriers for practitioners working with proprietary or sensitive datasets. This work eliminates both constraints through a data-free and training-free approach using channelwise parameter clustering, making it immediately applicable across diverse deployment scenarios.

The research builds on growing recognition that foundation models are overparameterized for many tasks. Recent advances in structured pruning and parameter clustering have shown promise, but this work uniquely combines these techniques with layer-level mixed sparsity, allowing different network sections to maintain varying compression ratios. This flexibility is crucial since different layers contribute differently to model performance, and uniform pruning often wastes compression capacity.

The experimental results demonstrate substantial practical value. On HuBERT-large at 50% sparsity, the method outperforms magnitude-based pruning by 34.37% relative WER reduction before fine-tuning, with modest additional gains after brief fine-tuning. Whisper-large-v3 shows even more dramatic improvements at 10% sparsity. These results matter for edge deployment, where reduced model size directly translates to lower latency, memory consumption, and inference costs.

For developers and organizations deploying speech AI systems, this technique offers immediate benefits without architectural redesign or data exposure risks. The approach could accelerate adoption of sophisticated speech models in resource-constrained environments, particularly in privacy-sensitive applications where data cannot leave device boundaries. Future work likely involves extending these methods to other foundation model architectures and exploring adaptive clustering strategies.

• →Data-free and training-free compression eliminates major deployment barriers for speech foundation models
• →Layer-level mixed sparsity outperforms uniform pruning, achieving 27-59% relative WER improvements
• →Method requires only brief fine-tuning (3 epochs) to achieve near-baseline performance
• →Approach maintains compatibility with existing models without architectural modifications
• →Enables efficient speech AI deployment on edge devices and privacy-sensitive applications