How LoRA Remembers? A Parametric Memory Law for LLM Finetuning

Researchers introduce the Parametric Memory Law, a power law framework quantifying how Large Language Models store information through Low-Rank Adaptation (LoRA) finetuning. The study reveals a deterministic phase transition at the token level and proposes MemFT, an optimization strategy that improves memory fidelity by dynamically redistributing training resources toward undertrained tokens.

This research addresses a fundamental gap in understanding how LLMs encode and retain information during finetuning. While LoRA has become the industry standard for parameter-efficient adaptation, the quantitative mechanics of memory capacity have remained largely opaque. By treating LoRA as a controlled memory probe, researchers establish measurable relationships between loss reduction, parameter count, and sequence length—moving beyond qualitative downstream benchmarks toward predictive, mathematical models of learning dynamics.

The discovery of a deterministic phase transition at p > 0.5 prediction probability marks a critical threshold for verbatim recall under greedy decoding. This finding has practical implications: it identifies which tokens require additional training emphasis and which have already achieved reliable memorization. The Parametric Memory Law, expressed as a power law, provides a unified framework for predicting memory behavior across different model scales and configurations.

For AI practitioners, these insights enable more efficient training protocols. MemFT's threshold-guided approach reduces computational waste by targeting optimization effort toward genuinely undertrained tokens rather than uniformly distributing resources. This efficiency gain matters significantly in production environments where finetuning costs scale with model size and dataset complexity. The methodology also provides foundation builders and researchers with quantitative tools to diagnose memory bottlenecks in their pipelines.

Longer-term implications suggest this framework could guide architecture decisions and inform decisions about when to scale parameters versus refining training algorithms. As LLMs require increasingly frequent knowledge updates in dynamic domains, understanding these memory dynamics becomes competitive advantage for deploying cost-effective, reliable models.

• →The Parametric Memory Law establishes a robust power law relationship between loss reduction, parameter count, and sequence length in LLM finetuning.
• →A phase transition occurs when prediction probability exceeds 0.5, marking the threshold for reliable verbatim token recall.
• →MemFT dynamically redistributes training budget toward sub-threshold tokens, improving memory efficiency without additional parameters.
• →Quantitative memory analysis reveals that LoRA finetuning follows deterministic dynamics measurable across different model scales.
• →This framework enables more cost-effective knowledge updates for LLMs operating in dynamic environments.