SiNGER: A Clearer Voice Distills Vision Transformers Further

Researchers introduce SiNGER, a new knowledge distillation framework for Vision Transformers that suppresses harmful high-norm artifacts while preserving informative signals. The technique uses nullspace-guided perturbation and LoRA-based adapters to achieve state-of-the-art performance in downstream tasks.

• →Vision Transformers produce high-norm artifacts that degrade representation quality and hinder knowledge distillation effectiveness.
• →SiNGER framework addresses the trade-off between artifact suppression and signal preservation in teacher-student model training.
• →The method uses nullspace-guided perturbation with LoRA-based adapters requiring minimal structural modifications.
• →Extensive experiments demonstrate consistent improvements in student models across multiple downstream tasks.
• →The approach produces clearer and more interpretable representations compared to existing distillation methods.