AIBullisharXiv – CS AI · 18h ago7/10
🧠COLLEAGUE.SKILL is an open-source system that automates the conversion of expert knowledge traces into portable, inspectable AI agent skills through a structured distillation workflow. The framework enables person-grounded agents to encode human expertise, decision-making patterns, and communication styles as versioned, correctable skill packages that can be deployed across multiple agent hosts.
AIBullisharXiv – CS AI · 3d ago7/10
🧠LoopFM introduces a novel knowledge distillation framework that transfers rich intermediate representations from large foundation models to compact vertical models, achieving significant conversion improvements (0.5-1.22%) in industrial-scale systems by structuring FM embeddings as input features rather than relying on single scalar predictions.
AIBullisharXiv – CS AI · 3d ago7/10
🧠Researchers introduce CORE-RAG, a novel framework that compresses context in Retrieval-Augmented Generation systems using performance-driven learning rather than predefined heuristics. The approach achieves a 97% compression ratio while improving accuracy by 3.3 points on exact match scores, addressing a critical bottleneck in LLM efficiency.
AIBullisharXiv – CS AI · 4d ago7/10
🧠Researchers propose LIFT and PLACE, a knowledge distillation framework that enables stable training of extremely lightweight diffusion models by decomposing the teacher's complex denoising process into coarse and fine stages with spatially adaptive guidance. The method achieves stable convergence even at extreme compression ratios (1.6% of teacher size) where conventional distillation fails, with potential applications across image generation, latent diffusion, and flow-based models.
AIBullisharXiv – CS AI · 4d ago7/10
🧠Researchers present a framework for converting Mixture-of-Experts (MoE) language models into standard dense architectures through expert selection, grouping, and knowledge distillation. The method achieves superior performance compared to traditional dense-to-dense pruning while enabling deployment on memory-constrained systems.
AIBullisharXiv – CS AI · May 127/10
🧠Researchers introduce Mixed-Policy Distillation (MPD), a technique that compresses reasoning in smaller language models by having larger teacher models rewrite student-generated reasoning traces into more concise versions. The method reduces token usage by up to 27.1% while maintaining or improving performance, addressing critical deployment constraints around memory, latency, and serving costs.
AIBullisharXiv – CS AI · May 127/10
🧠MedThink presents a two-stage knowledge distillation framework that improves diagnostic accuracy in smaller language models by having teacher LLMs guide reasoning correction rather than simply transferring surface-level patterns. The approach achieves up to 12.7% improvement over baseline models while maintaining computational efficiency for resource-constrained clinical environments.
AIBullisharXiv – CS AI · May 127/10
🧠Researchers introduce LiteMedCoT-VL, a technique that transfers chain-of-thought reasoning from large language models to compact 2B parameter models for medical visual question answering, achieving 64.9% accuracy on the PMC-VQA benchmark without relying on image captions. The breakthrough demonstrates that smaller models enhanced with reasoning distillation can match or exceed the performance of larger models, enabling deployment of sophisticated medical AI on resource-constrained clinical devices.
AIBullisharXiv – CS AI · May 127/10
🧠Researchers present SlimQwen, a systematic study of compression techniques for mixture-of-experts (MoE) language models during pretraining. The work demonstrates that pruning pretrained MoE models outperforms training smaller architectures from scratch, and proposes progressive pruning combined with knowledge distillation as the most effective compression strategy, successfully compressing Qwen3-Next-80A3B to 23A2B while maintaining competitive performance.
AIBullisharXiv – CS AI · May 77/10
🧠EdgeRazor introduces a lightweight quantization framework that compresses large language models to 1.88-bit precision while maintaining performance superior to existing 3-bit methods. The approach combines mixed-precision quantization with knowledge distillation and achieves up to 15.1× faster decoding with 80% storage reduction, requiring significantly lower computational training budgets than comparable techniques.
AIBullisharXiv – CS AI · Mar 177/10
🧠Researchers developed Token-Selective Dual Knowledge Distillation (TSD-KD), a new framework that improves AI reasoning by allowing smaller models to learn from larger ones more effectively. The method achieved up to 54.4% better accuracy than baseline models on reasoning benchmarks, with student models sometimes outperforming their teachers by up to 20.3%.
AIBullisharXiv – CS AI · Mar 57/10
🧠Researchers from KAIST propose AMiD, a new knowledge distillation framework that improves the efficiency of training smaller language models by transferring knowledge from larger models. The technique introduces α-mixture assistant distribution to address training instability and capacity gaps in existing approaches.
AIBullisharXiv – CS AI · Mar 46/103
🧠Researchers introduce SiNGER, a new knowledge distillation framework for Vision Transformers that suppresses harmful high-norm artifacts while preserving informative signals. The technique uses nullspace-guided perturbation and LoRA-based adapters to achieve state-of-the-art performance in downstream tasks.
AIBullisharXiv – CS AI · Mar 46/102
🧠Researchers propose Router Knowledge Distillation (Router KD) to improve retraining-free compression of Mixture-of-Experts (MoE) models by calibrating routers while keeping expert parameters unchanged. The method addresses router-expert mismatch issues that cause performance degradation in compressed MoE models, showing particularly strong results in fine-grained MoE architectures.
AINeutralarXiv – CS AI · 18h ago5/10
🧠Researchers propose Trust-Region behavior Blending (TRB), a warmup technique that improves on-policy distillation by having student models learn from a teacher-aligned policy during early training stages rather than weak student rollouts. The method anneals the constraint over time until training returns to pure student policy, demonstrating stronger performance in math-reasoning tasks.
AINeutralarXiv – CS AI · 18h ago6/10
🧠Researchers introduce MIMO, a two-stage framework for multilingual information retrieval that leverages monolingual objectives to improve cross-lingual search performance. By using knowledge distillation from a high-performing English model and combining it with cross-lingual contrastive learning, MIMO addresses the language clustering problem that degrades existing embedding models in mixed-language retrieval scenarios.
AINeutralarXiv – CS AI · 3d ago5/10
🧠TaxDistill introduces a knowledge distillation framework using GenomeOcean, a 500M-parameter genomic foundation model, to improve metagenomic taxonomic annotation by reducing label noise from sequence similarity tools. The approach achieves significant performance gains, improving F1 scores by 23.3% on gastrointestinal datasets compared to traditional methods.
AIBullisharXiv – CS AI · 3d ago6/10
🧠Researchers propose Canonical-Context On-Policy Distillation (CCOPD), a training method that improves large language models' ability to solve problems when information is revealed incrementally across multiple conversation turns rather than all at once. By using a frozen teacher model with complete context to guide a student model receiving fragmented information, CCOPD achieves 32% relative performance improvement on multi-turn tasks while maintaining single-prompt performance.
AIBullisharXiv – CS AI · 3d ago6/10
🧠Researchers propose REKD (Rationale Extraction with Knowledge Distillation), a method that improves the interpretability and performance of smaller deep neural networks by having them learn from larger teacher models' rationales and predictions. The approach demonstrates significant performance gains across language and vision tasks, offering a practical framework for making AI systems more transparent and verifiable in high-stakes applications.
AINeutralarXiv – CS AI · 3d ago6/10
🧠Researchers introduce eXTC, a new framework combining structured prompt optimization with reinforcement learning to create interpretable text classifiers that balance performance with explainability. The system generates human-readable domain rules while maintaining inference speed through knowledge distillation, addressing a longstanding trade-off in AI transparency.
AINeutralarXiv – CS AI · 3d ago5/10
🧠Researchers introduce xModel-KD, a cross-modal knowledge distillation framework that combines 2D image data with 3D LiDAR point clouds to improve 3D scene segmentation with fewer labeled examples. The method achieves 2% absolute mIoU improvement over LiDAR-only approaches by leveraging complementary strengths of texture and geometric information through contrastive learning.
AIBullisharXiv – CS AI · 4d ago6/10
🧠Researchers demonstrate that a 0.6B-parameter ASR model trained on 100k hours of speech can achieve competitive performance with larger models through teacher-guided on-policy distillation, reducing the audio data requirements by 99.5% compared to industry standards while closing the capability gap with 1.7B parameter models.
AIBullisharXiv – CS AI · 4d ago6/10
🧠Researchers propose entropy-aware masking for masked language modeling, which selectively masks tokens based on prediction uncertainty rather than random selection. The approach achieves 5% improvement in GLUE scores and performs best when combined with knowledge distillation, offering a more efficient pretraining strategy for encoder-based language models.
AIBullisharXiv – CS AI · 4d ago6/10
🧠Researchers introduce STARS, a data-free knowledge distillation method that improves the transfer of learning from artificial neural networks (ANNs) to spiking neural networks (SNNs) without access to original training data. The technique combines batch normalization matching with relational consistency and threshold-aware regularization, achieving significant accuracy improvements across standard benchmarks.
AINeutralarXiv – CS AI · 4d ago6/10
🧠Researchers introduce Multi-Teacher Bayesian Knowledge Distillation (MT-BKD), a framework that enables student models to learn from multiple teacher models while quantifying uncertainty through Bayesian inference. The approach uses teacher-informed priors and entropy-based weighting to improve model compression, generalization, and interpretability across synthetic and real-world tasks.
