AMiD: Knowledge Distillation for LLMs with $\alpha$-mixture Assistant Distribution

Researchers from KAIST propose AMiD, a new knowledge distillation framework that improves the efficiency of training smaller language models by transferring knowledge from larger models. The technique introduces α-mixture assistant distribution to address training instability and capacity gaps in existing approaches.

• →AMiD introduces α-mixture assistant distribution as a generalized framework for knowledge distillation in large language models.
• →The approach addresses fundamental limitations including capacity gaps and training instability caused by near-zero probabilities in high-dimensional LLM outputs.
• →The framework provides a continuous extension of assistant distributions through a new design variable α that was previously fixed in other methods.
• →Extensive experiments demonstrate superior performance and training stability compared to existing knowledge distillation approaches.
• →The research offers a unified theoretical framework that generalizes previous fragmented approaches to assistant distributions.