From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression

Researchers introduce SubFit, a post-training compression method for Large Language Models that operates at the submodule level rather than full-layer granularity, achieving superior perplexity-accuracy trade-offs. The approach selects non-contiguous Attention and FeedForward submodules with individual fitted residual bypasses, delivering 84.6% downstream accuracy retention at 25% sparsity compared to 81.6% for existing methods.

SubFit addresses a fundamental limitation in current LLM compression techniques by challenging the assumption that redundancy clusters in contiguous architectural layers. Traditional replacement-based compression methods treat entire layers as atomic units, but this research demonstrates that redundancy distributes unevenly across Attention and FeedForward components, requiring differentiated compression strategies. By decomposing layers into submodules and enabling non-contiguous selection, SubFit achieves substantially better compression efficiency.

The work builds on growing recognition that transformer architectures contain significant redundancy amenable to removal without catastrophic performance loss. Previous compression approaches—including pruning, quantization, and layer replacement—have improved inference efficiency but sacrificed accuracy more severely than SubFit. The technical contribution lies in identifying that different submodule types respond differently to compression, enabling targeted rather than blanket removal strategies.

For practitioners and model deployers, SubFit's practical benefits are significant: measurable inference speedup, reduced KV-cache memory requirements, and minimal accuracy degradation enable efficient deployment of LLMs on resource-constrained hardware. The method's post-training application without fine-tuning makes it immediately applicable to existing models. At 25% sparsity, SubFit's 2.42x perplexity degradation substantially outperforms baseline 4.34x degradation, representing meaningful efficiency gains for production systems.

The research opens opportunities for further granularity exploration—potentially extending to layer-internal components or attention head levels. As LLM inference costs remain a critical deployment bottleneck, compression techniques enabling efficient serving directly impact model accessibility and operational economics across organizations scaling AI infrastructure.

• →SubFit achieves 84.6% downstream accuracy retention at 25% sparsity versus 81.6% for comparable methods through submodule-level granularity
• →Non-contiguous submodule selection with fitted residual bypasses outperforms traditional full-layer compression approaches across multiple models and sparsity levels
• →Post-training compression requiring only calibration data enables immediate application to existing pretrained LLMs without additional training
• →Method delivers measurable inference speedup and KV-cache memory savings, addressing key deployment constraints for resource-limited environments
• →Redundancy distribution analysis reveals Attention and FeedForward submodules require differentiated compression strategies for optimal efficiency gains