IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoring

Researchers introduce IGU-LoRA, a new parameter-efficient fine-tuning method for large language models that adaptively allocates ranks across layers using integrated gradients and uncertainty-aware scoring. The approach addresses limitations of existing methods like AdaLoRA by providing more stable and accurate layer importance estimates, consistently outperforming baselines across diverse tasks.

• →IGU-LoRA uses integrated gradients to better capture layer importance compared to instantaneous gradient methods like AdaLoRA.
• →The method incorporates uncertainty-aware scoring with exponential moving averages to reduce noise in rank allocation decisions.
• →Theoretical analysis provides upper bounds on approximation error under pathwise Hessian-Lipschitz conditions.
• →Experimental results show consistent improvements in downstream accuracy and robustness across multiple tasks and architectures.
• →The approach addresses the compute and memory limitations of full-parameter fine-tuning for billion-parameter language models.