The Fine-Tuning Trap: Evaluating Negative Transfer and the Role of PEFT in Sub-1B Mathematical Reasoning

Researchers benchmarked five sub-1B language models and discovered that Full Fine-Tuning actively degrades performance on models under 300M parameters, causing accuracy to drop below zero-shot baselines. Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA and DoRA prove necessary for stability, with task-specific strengths that outperform full fine-tuning and sometimes even match in-context learning on the smallest architectures.

This research addresses a critical pain point in deploying small language models on edge devices: the counterintuitive failure of traditional fine-tuning approaches on sub-1B parameter models. The study reveals that Full Fine-Tuning causes catastrophic forgetting in models under 300M parameters, pushing accuracy below what untrained models achieve through zero-shot reasoning. This negative transfer effect fundamentally shifts how practitioners should approach model adaptation for resource-constrained environments.

The broader context reflects growing interest in efficient AI deployment as edge computing becomes increasingly important for latency-sensitive and privacy-focused applications. Full Fine-Tuning has historically dominated because it requires no architectural changes, but this research demonstrates that efficiency techniques like LoRA and DoRA aren't merely optimizations—they're stability requirements for smaller models. The finding that DoRA excels in complex reasoning while LoRA dominates pattern matching suggests task-specific selection becomes necessary rather than optional.

For developers and organizations deploying small models, this has immediate practical implications. The recommendation against Full Fine-Tuning for any architecture under 500M parameters provides clear guardrails that prevent wasteful experimentation. The discovery that 5-shot In-Context Learning occasionally matches fine-tuned performance on 135M models challenges assumptions about fine-tuning necessity itself. This opens possibilities for simpler deployment pipelines without quality degradation.

Looking ahead, researchers should investigate why negative transfer occurs at specific model scales and whether architectural improvements can mitigate these effects. The open-source reproduction materials enable community validation and extension, potentially establishing model-scaling thresholds as critical benchmarks for fine-tuning strategy selection.

• →Full Fine-Tuning degrades accuracy below zero-shot baselines on models under 300M parameters due to catastrophic forgetting.
• →PEFT methods like LoRA and DoRA are stability requirements rather than optional efficiency improvements for sub-1B models.
• →Task-specific strengths vary: DoRA excels in complex reasoning while LoRA dominates pattern matching tasks.
• →In-Context Learning sometimes matches fine-tuned performance on smallest architectures, simplifying deployment options.
• →Practitioners should avoid Full Fine-Tuning for any model smaller than 500M parameters to prevent performance degradation.