Finer Parameter Steps for Low-Rank PEFT: A Controlled Study with CP Tensor Adapters

Researchers compare canonical polyadic (CP) tensor adapters with LoRA for low-rank parameter-efficient fine-tuning, finding that finer parameter increments enable better budget sensitivity diagnosis but don't guarantee superior accuracy-budget trade-offs across all tasks.

This research addresses a fundamental limitation in parameter-efficient fine-tuning (PEFT) methods: the coarse granularity of rank selection in popular adapters like LoRA. When adapting large models, practitioners face discrete jumps in trainable parameters—increasing LoRA rank by one step adds thousands of scalars, creating gaps where optimal configurations may exist but remain inaccessible. The paper proposes CP tensor adapters, which allow much finer capacity increments of approximately 193 parameters per component compared to LoRA's ~4,096 per rank step, enabling researchers to precisely probe parameter-accuracy relationships.

The controlled experimental design strengthens the findings by holding training protocols, seed schedules, and target modules constant across OPT-1.3B benchmarks. Results reveal task-dependent behavior: SST-2 shows early saturation regardless of fine-grained parameter increments, BoolQ exploits additional capacity before plateauing slightly below LoRA performance, and RTE consistently favors LoRA. This variability suggests that fine parameter resolution alone cannot overcome fundamental efficiency differences between adapter architectures.

For the machine learning community, this work legitimizes tensor-based adapters as diagnostic tools for understanding PEFT budget sensitivity. Practitioners now have evidence that CP methods effectively fill parameter gaps, making them valuable for resource-constrained scenarios or exploratory phases. However, the neutral-to-slightly-negative accuracy outcomes compared to LoRA limit immediate adoption. The findings highlight that architectural choice interacts meaningfully with task properties, cautioning against universal claims about adapter superiority. Future work should investigate why certain tasks benefit from fine increments while others remain LoRA-dominant, potentially unlocking task-specific adapter design principles.

• →CP tensor adapters enable 21x finer parameter increments than LoRA, allowing precise budget sensitivity analysis
• →Fine-grained parameter steps improve diagnosis capabilities but do not guarantee better accuracy-budget curves
• →Task-dependent results show SST-2 saturates early, BoolQ benefits from additional capacity, and RTE favors LoRA
• →Controlled experimental conditions (matching protocols, seeds, modules) isolate adapter architecture effects from confounding factors
• →Fine parameter resolution's diagnostic value exceeds its performance improvements, reframing the contribution as methodological rather than empirical