On-the-Fly Adaptation to Quantization: Configuration-Aware LoRA for Efficient Fine-Tuning of Quantized LLMs

Researchers introduce CoA-LoRA, a method that dynamically adapts LoRA fine-tuning to different quantization configurations without requiring separate retraining for each setting. The approach uses a configuration-aware model and Pareto-based search to optimize low-rank adjustments across heterogeneous edge devices, achieving comparable performance to traditional methods with zero additional computational cost.

CoA-LoRA addresses a critical bottleneck in deploying large language models on resource-constrained devices. As quantization has emerged as the primary compression technique for edge deployment, the computational burden of fine-tuning separate LoRA adapters for each quantization configuration has become prohibitively expensive. This research tackles that inefficiency by creating a single model capable of predicting optimal low-rank adjustments for arbitrary bit-width configurations.

The technical innovation centers on two components: a configuration-aware neural network that maps quantization settings to their corresponding LoRA parameters, and a Pareto-based search algorithm that intelligently selects training configurations. Rather than exhaustively fine-tuning for every possible bit-width combination, this approach learns generalizable patterns across a carefully curated subset. The Pareto optimization ensures the training set balances coverage across different bit-width budgets, maximizing the effectiveness of the learned mappings.

For practitioners deploying LLMs in edge computing scenarios, this represents meaningful progress toward practical on-device inference. Edge devices exhibit heterogeneous hardware—varying memory, compute, and power constraints—making the ability to dynamically adapt to different quantization levels without retraining highly valuable. The method eliminates what was previously a manual, computationally expensive step in the deployment pipeline.

The implications extend to privacy-preserving applications where keeping inference local is essential. By reducing deployment friction and computational overhead, CoA-LoRA enables faster iteration cycles for organizations deploying quantized models across diverse hardware environments. Future work likely focuses on extending this approach to other compression techniques beyond quantization.

• →CoA-LoRA dynamically adapts LoRA adapters to quantization configurations without retraining, eliminating significant computational overhead
• →Configuration-aware models combined with Pareto-based search enable generalization across heterogeneous quantization settings
• →Method achieves performance comparable to or exceeding traditional per-configuration fine-tuning approaches at zero additional cost
• →Approach accelerates deployment of quantized LLMs on edge devices with varying hardware capabilities
• →Reduces barriers to privacy-preserving on-device inference by streamlining the quantization-adaptation pipeline