Self-Distillation as a Performance Recovery Mechanism for LLMs: Counteracting Compression and Catastrophic Forgetting

Researchers introduce Self-Distillation Fine-Tuning (SDFT), a framework that recovers performance degradation in Large Language Models caused by compression, quantization, and catastrophic forgetting. Using Centered Kernel Alignment analysis, the study demonstrates that self-distillation works by aligning the student model's high-dimensional manifold with the teacher model's optimal representation structure.

This arXiv paper addresses a critical practical challenge in LLM deployment: performance loss during fine-tuning, quantization, and model compression. These operations are essential for making LLMs cost-effective and deployable at scale, but they consistently degrade model capabilities. The proposed self-distillation framework offers a principled approach to recovery that moves beyond empirical patching toward theoretical understanding.

The research builds on established knowledge that neural networks encode information in high-dimensional manifolds within their hidden layers. By employing Centered Kernel Alignment—a tool specifically designed to handle invariances in neural representations—the authors quantify how well student and teacher models align at the representation level. This geometric perspective explains why self-distillation works: it doesn't just imitate outputs, but reconstructs the underlying representational structure that enables generative capability.

For the LLM industry, this has meaningful implications. As models grow larger and deployment demands require compression and pruning, recovery mechanisms become crucial infrastructure. Rather than accepting performance degradation as inevitable, practitioners can apply SDFT to restore capabilities systematically. This reduces the trade-off between model efficiency and capability, making high-performance smaller models more feasible.

The bridging of practical and theoretical perspectives creates a foundation for more sophisticated model optimization strategies. Future work may leverage this manifold-alignment insight to design better compression techniques upfront or develop adaptive fine-tuning methods. Teams building production LLM systems should monitor whether SDFT becomes standard practice in model optimization pipelines.

• →Self-distillation effectively recovers LLM performance degraded by quantization, pruning, and catastrophic forgetting during fine-tuning.
• →The recovery mechanism works by aligning the student model's high-dimensional representation manifold with the teacher model's structure.
• →Centered Kernel Alignment provides a geometric framework to measure and explain self-distillation effectiveness empirically.
• →The approach bridges practical model optimization with representation learning theory, enabling more principled compression strategies.
• →Results suggest that manifold alignment, rather than mere output imitation, is the key to successful knowledge transfer in distillation.