Locality-Aware Redundancy Pruning for LLM Depth Compression

Researchers propose Locality-Aware Redundancy Pruning (LoRP), a training-free method for compressing large language models by removing redundant layers based on representational similarity patterns. The framework uses a Representation Locality Score to identify and prune depth-wise redundancy more effectively than existing approaches, improving both perplexity and downstream task performance across multiple LLM architectures.

LoRP addresses a critical efficiency challenge in large language models by recognizing that redundancy patterns vary significantly across different architectures. Rather than applying uniform pruning strategies, the method identifies how representational redundancy clusters within specific models—some concentrate redundancy locally while others distribute it globally. This architectural awareness represents a meaningful refinement in model compression techniques.

The research builds on established understanding that LLMs contain significant representational waste across their depth, but previous one-shot pruning methods failed to account for architecture-specific redundancy distribution. By computing inter-layer hidden-state similarity and clustering layers with similar representations, LoRP enables more targeted removal of truly redundant layers rather than indiscriminate depth reduction.

For practitioners deploying LLMs, this matters substantially. Model compression directly impacts inference latency, memory requirements, and computational costs—factors that determine real-world feasibility for edge deployment, mobile applications, and resource-constrained environments. Training-free methods like LoRP prove particularly valuable because they avoid expensive fine-tuning while maintaining model performance.

The breakthrough lies in combining simplicity with sophistication: using only a small calibration set while respecting architectural nuances produces measurable improvements in both standardized metrics and practical task performance. As LLM deployment becomes increasingly cost-sensitive and latency-critical, methods enabling efficient model compression without retraining gain competitive importance across inference providers and edge AI applications.

• →LoRP introduces Representation Locality Score to map redundancy patterns that vary across different LLM architectures rather than assuming uniform compression needs.
• →Training-free pruning approach eliminates expensive fine-tuning while improving both perplexity metrics and downstream task accuracy across diverse model families.
• →Method identifies that inter-layer redundancy concentrates locally in some architectures but distributes globally in others, enabling tailored compression strategies.
• →Small calibration sets enable practical implementation for practitioners seeking to compress models without full retraining pipelines.
• →Advances in depth pruning directly reduce inference costs and latency, making efficient LLM deployment more viable for resource-constrained environments.