Rejuvenating Cross-Entropy Loss in Knowledge Distillation for Recommender Systems

Researchers propose Rejuvenated Cross-Entropy for Knowledge Distillation (RCE-KD) to improve knowledge distillation in recommender systems by addressing limitations of Cross-Entropy loss when distilling teacher model rankings. The method splits teacher's top items into subsets and uses adaptive sampling to better align with theoretical assumptions.

• →Cross-Entropy loss in knowledge distillation for recommenders only maximizes NDCG lower bound under specific closure assumptions that often aren't met.
• →There's a significant gap between items ranked highly by teacher models versus student models in recommender systems.
• →RCE-KD splits teacher's top items into subsets based on student rankings and uses collaborative sampling to bridge the gap.
• →The proposed method combines losses from different subsets adaptively to improve knowledge distillation effectiveness.
• →Extensive experiments demonstrate the effectiveness of RCE-KD over traditional cross-entropy approaches in recommender systems.