UniRank: Unified Rank Allocation for Low-Rank LLM Compression

Researchers propose UniRank, a new method for efficiently allocating ranks in low-rank decomposition of large language models by scoring components via local singular energy and global functional importance. The approach achieves up to 50% perplexity reduction compared to baseline methods without additional fine-tuning, addressing a key bottleneck in LLM compression.

UniRank tackles a fundamental challenge in neural network compression: determining which parameters matter most when reducing model size through low-rank decomposition. Traditional approaches either rely on hand-crafted rules that don't generalize across different architectures or employ computationally expensive learning-based methods. This research introduces a two-pronged scoring mechanism that evaluates both the intrinsic mathematical importance of decomposed components and their functional significance measured through input-output similarity.

The compression landscape has evolved significantly as model sizes grew unwieldy. Low-rank decomposition emerged as a promising alternative to pruning and quantization because it preserves model structure while reducing parameters. However, the critical question of where to allocate limited rank capacity remained largely unresolved. UniRank's insight—that input-output cosine similarity correlates with low effective rank—provides both theoretical grounding and practical utility.

The method's integration of rank-preserving fine-tuning through direct LoRA application represents a meaningful improvement over conventional merging pipelines that incur information loss. Empirical validation across diverse model architectures and sizes strengthens the approach's generalizability claim. For practitioners developing efficient LLMs, this addresses a genuine pain point: deploying models on resource-constrained devices without extensive retraining.

The availability of open-source code accelerates adoption among researchers and engineers. The performance gains—particularly the one-shot compression results—suggest UniRank could become standard in the LLM compression toolkit. As models continue growing, methods that efficiently balance performance and computational efficiency gain increasing strategic importance in making AI more accessible.

• →UniRank uses dual scoring criteria combining local singular energy and global functional importance to optimize rank allocation in LLM compression.
• →The method achieves up to 50% perplexity reduction in one-shot compression without additional fine-tuning compared to uniform baselines.
• →Input-output cosine similarity correlates strongly with low effective rank, providing both theoretical validation and practical guidance for rank allocation.
• →Rank-preserving LoRA fine-tuning avoids information loss from re-truncation in traditional model merging approaches.
• →The approach demonstrates generalizability across different decomposition schemes, model sizes, and architectural designs.