Polynomial Expansion Rank Adaptation: Enhancing Low-Rank Fine-Tuning with High-Order Interactions

Researchers propose Polynomial Expansion Rank Adaptation (PERA), a novel fine-tuning method that enhances Low-Rank Adaptation (LoRA) by incorporating high-order polynomial interactions into low-rank factors. PERA improves the expressive capacity of LLM fine-tuning without increasing computational costs, demonstrating consistent performance gains across benchmarks while maintaining the efficiency benefits of rank-constrained adaptation.

PERA addresses a fundamental limitation in current LLM fine-tuning practices. LoRA has become the dominant approach for efficient parameter adaptation because it dramatically reduces the number of trainable parameters while maintaining strong performance. However, its strictly linear structure constrains the model's ability to capture complex parameter interactions, forcing practitioners to choose between expressive power and computational efficiency.

The innovation lies in introducing polynomial expansion directly within the low-rank factor space rather than at the weight level. By synthesizing high-order interaction terms—particularly quadratic terms—before composition, PERA creates a richer representation space without the computational overhead typically associated with higher-dimensional adaptation. This transforms the adaptation landscape into a polynomial manifold capable of modeling nonlinear dependencies that standard LoRA cannot express.

For the AI development community, PERA represents meaningful progress in efficient fine-tuning technology. As LLMs continue growing in size, techniques that improve adaptation quality without proportionally increasing compute costs become increasingly valuable. The theoretical analysis backing PERA suggests this isn't merely incremental improvement but a fundamental enhancement to how rank-constrained adaptation operates. Developers can achieve better results with equivalent or smaller rank settings, directly reducing memory requirements and inference latency.

The practical implications extend across production deployments where computational constraints matter. Organizations implementing domain-specific LLM variants benefit from maintaining low-rank efficient architectures while accessing improved model capability. The open-source release enables rapid adoption and further research validation across diverse applications, potentially influencing how future efficient fine-tuning methods are designed.

• →PERA enhances LoRA by incorporating polynomial expansion to capture high-order parameter interactions without increasing inference cost
• →The method demonstrates consistent performance improvements across multiple benchmarks compared to existing linear adaptation approaches
• →Quadratic terms prove particularly important for improving expressive capacity and maintaining robustness across different rank settings
• →PERA maintains computational efficiency while providing richer nonlinear modeling capabilities for parameter adaptation
• →Open-source availability enables rapid community adoption and validation of the polynomial expansion approach