AIBullisharXiv – CS AI · Jun 97/10
🧠Researchers introduced ZEDA, a framework that converts fully-trained Mixture-of-Experts language models into dynamic variants capable of skipping unnecessary experts, reducing computational requirements by over 50% with minimal accuracy loss. The method uses self-distillation to adapt post-trained models without retraining from scratch, achieving ~1.20x end-to-end inference speedup on major language models.
AIBullisharXiv – CS AI · Jun 47/10
🧠Researchers propose Distribution Aligned Imitation Learning (DAIL), a self-distillation method that improves LLM reasoning by converting expert human solutions into computational training data. The technique achieves significant performance gains on frontier models using fewer than 1000 expert examples, addressing the challenge that expert solutions are typically written for humans rather than machines.
AIBullisharXiv – CS AI · Jun 17/10
🧠Researchers propose Feedback Distillation, a novel post-training method for language models that improves reasoning tasks by having models learn from their own feedback at the token level. Applied to Lean4 theorem-proving, the approach outperforms standard GRPO methods in trajectory diversity and scalability while complementing existing reinforcement learning approaches.
AIBullisharXiv – CS AI · May 297/10
🧠Researchers propose Guided Denoiser Self-Distillation (GDSD), a new reinforcement learning method for diffusion language models that eliminates the need for evidence lower bound approximations, achieving up to 19.6% performance improvements over existing approaches on planning, math, and coding tasks.
AIBullisharXiv – CS AI · May 277/10
🧠Search-E1 introduces a simplified self-evolution method for search-augmented reasoning agents that achieves competitive performance through vanilla GRPO and self-distillation, without external supervision or complex auxiliary systems. The approach reaches 0.440 average EM on QA benchmarks with Qwen2.5-3B, demonstrating that elaborate post-training machinery may be unnecessary for effective agent development.
AINeutralarXiv – CS AI · Apr 207/10
🧠Researchers identify that supervised fine-tuning of large language models increases hallucinations by degrading pre-existing knowledge through semantic interference. The study proposes self-distillation and parameter freezing techniques to mitigate this problem while preserving task performance.
AIBullisharXiv – CS AI · Apr 137/10
🧠SkillFactory is a novel fine-tuning method that enables language models to learn cognitive behaviors like verification and backtracking without requiring distillation from stronger models. The approach uses self-rearranged training samples during supervised fine-tuning to prime models for subsequent reinforcement learning, resulting in better generalization and robustness.
AIBullisharXiv – CS AI · Mar 277/10
🧠Researchers developed GoldiCLIP, a data-efficient vision-language model that achieves state-of-the-art performance using only 30 million images - 300x less data than leading methods. The framework combines three key innovations including text-conditioned self-distillation, VQA-integrated encoding, and uncertainty-based loss weighting to significantly improve image-text retrieval tasks.
AIBullisharXiv – CS AI · Mar 267/10
🧠Researchers propose MTP-D, a self-distillation method that improves Multi-Token Prediction for Large Language Models, achieving 7.5% better acceptance rates and up to 220% inference speedup. The technique addresses key challenges in training multiple prediction heads while preserving main model performance.
AIBullisharXiv – CS AI · Mar 167/10
🧠Researchers developed a new method for training AI language models using multi-turn user conversations through self-distillation, leveraging follow-up messages to improve model alignment. Testing on real-world WildChat conversations showed improvements in alignment and instruction-following benchmarks while enabling personalization without explicit feedback.
AIBearisharXiv – CS AI · Jun 256/10
🧠Researchers reveal that on-policy self-distillation, a technique that improves single-model accuracy by using correct demonstrations as conditioning, reduces output diversity and flattens pass@k curves—meaning additional rollouts fail to boost performance. The method amplifies existing model biases rather than preserving probability ratios like optimal reinforcement learning does, causing models to concentrate on dominant modes and fail in out-of-distribution settings.
AINeutralarXiv – CS AI · Jun 116/10
🧠Researchers introduce HERO, a self-distillation framework for reinforcement learning agents that uses environment observations as feedback to improve multi-turn decision-making. The method addresses credit assignment problems in sequential tasks by converting observations into actionable diagnoses, outperforming existing approaches on benchmark tasks with limited training data.
AIBullisharXiv – CS AI · Jun 106/10
🧠Researchers introduce Visual-SDPO, a self-distillation framework that enables code-generating LLMs to improve visual artifact quality by learning from rendered output feedback. The method achieves 10+ point improvements on code-to-visual generation benchmarks while maintaining inference efficiency.
AINeutralarXiv – CS AI · Jun 106/10
🧠Researchers demonstrate that self-distillation in language models improves significantly when feedback is structurally aligned with the model's reasoning trace rather than using binary rewards or reference solutions. Step-aligned critique, which targets only tokens where reasoning fails, outperforms alternative approaches by 5-16 points, suggesting that feedback design fundamentally shapes model learning efficiency.
AINeutralarXiv – CS AI · Jun 56/10
🧠Researchers introduce MGSD, a self-distillation framework that improves vision-language models' ability to perform visual spatial planning by using symbolic state data during training to bridge the perception-reasoning gap. The approach achieves 18-19% performance improvements on visual planning benchmarks while maintaining purely visual inference.
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers propose CAST, a new self-distillation method for reinforcement learning in large language models that improves upon existing approaches by using answer-free teacher scoring and bidirectional advantage flipping. The method addresses limitations in Group Relative Policy Optimization (GRPO) by providing denser token-level guidance while maintaining alignment with trajectory correctness, demonstrating improvements in mathematical reasoning tasks.
AINeutralarXiv – CS AI · May 296/10
🧠Researchers introduce OISD, a new reinforcement learning framework that improves language model reasoning by having the final layer act as an internal teacher to guide intermediate layers through logit and attention alignment. The method demonstrates consistent improvements across mathematical reasoning tasks without requiring external data.
AIBullisharXiv – CS AI · May 296/10
🧠Researchers introduce TRACER, a novel finetuning method for multimodal AI models that addresses catastrophic forgetting and out-of-distribution robustness degradation. By replacing standard Exponential Moving Average teachers with Weighted Moving Average teachers and combining contrastive learning with multi-perspective distillation, the approach demonstrates consistent performance gains across CLIP backbone architectures without hyperparameter sensitivity.
AINeutralarXiv – CS AI · May 286/10
🧠Researchers propose SC-SDPO, an improved machine learning technique that enhances how large language models learn from their own feedback during training. By weighting training examples based on question difficulty, the method achieves 3-4% performance gains on reasoning benchmarks while maintaining stable training dynamics.
AIBullisharXiv – CS AI · May 286/10
🧠Researchers propose Skill-Conditioned Gated Self-Distillation (SGSD), a novel method for improving large language model reasoning by leveraging an experience-derived skill bank rather than trusted reference answers. The approach validates skills through a multi-teacher framework and demonstrates consistent improvements over existing methods on mathematical reasoning benchmarks.
AINeutralarXiv – CS AI · May 286/10
🧠Researchers introduce Vision-OPD, a self-distillation framework that improves multimodal large language models' ability to detect fine-grained visual details by training full-image models to match the performance of crop-focused models. The technique achieves competitive results against larger models without requiring external teachers, labels, or inference-time tools, addressing a critical weakness in current MLLMs.
AINeutralarXiv – CS AI · May 116/10
🧠Researchers introduce SHRED, a machine unlearning method for large language models that removes memorized private or copyrighted data without requiring a curated retain set of examples. By selectively demoting logits of high-information tokens while preserving model utility through self-distillation, SHRED achieves superior trade-offs between forgetting efficacy and performance compared to existing retain-set-dependent approaches.
AINeutralarXiv – CS AI · May 116/10
🧠Researchers propose Multilingual Self-Distillation (MSD), a framework that transfers safety safeguards from high-resource languages like English to vulnerable low-resource languages in large language models. The method eliminates the need for expensive multilingual response data by leveraging an LLM's existing safety capabilities, demonstrating effective cross-lingual protection across diverse jailbreak benchmarks.
AINeutralarXiv – CS AI · May 96/10
🧠Researchers demonstrate that On-Policy Self-Distillation (OPSD) functions primarily as a compression mechanism rather than a correction tool for thinking-enabled mathematical reasoning models. They propose a revised training pipeline (SFT → RLVR → OPSD) that leverages OPSD's strengths in shortening responses while preserving accuracy on correct outputs.
AIBullisharXiv – CS AI · May 96/10
🧠Researchers introduce UniSD, a unified self-distillation framework that systematically improves large language model adaptation without requiring external teacher models. The framework combines multiple complementary mechanisms and demonstrates consistent performance gains of +5.4 points over baseline models across six benchmarks, advancing efficient LLM training techniques.
