AIBullisharXiv – CS AI · 3d ago7/10
🧠Researchers introduce K-Forcing, a novel language modeling approach that enables autoregressive models to generate multiple tokens simultaneously rather than sequentially, achieving 2.4-3.5x inference speedup. The technique distills existing AR models into a push-forward mapping trained via progressive self-forcing, maintaining compatibility with standard serving infrastructure while trading modest quality for significant computational efficiency gains critical for industrial-scale LLM deployment.
AIBullisharXiv – CS AI · 3d ago7/10
🧠FADA is a unified vision-language model that performs fetal ultrasound interpretation, detection, and segmentation through a single pipeline, addressing critical diagnostic gaps in low- and middle-income countries where sonographer shortages limit prenatal screening. The system runs on consumer hardware and smartphones entirely offline, achieving clinically validated performance metrics while requiring no external labels at inference.
AIBullisharXiv – CS AI · Jun 57/10
🧠Researchers propose CKA-QAD, a new method for quantizing large language models to NVFP4 precision that preserves internal representational geometry rather than just matching output distributions. The approach addresses a critical limitation in existing quantization-aware distillation techniques, showing significant improvements in reasoning and coding task performance across multiple model architectures.
AIBullisharXiv – CS AI · Jun 57/10
🧠Researchers and LinkedIn introduce SAGE, a framework that combines human judgment with AI surrogates to evaluate search relevance at scale. By using a bidirectional calibration loop between policy, precedent examples, and LLM judges, the system achieves near-human agreement while reducing inference costs by 92×, ultimately driving a 0.25% lift in LinkedIn's daily active users.
AIBullisharXiv – CS AI · Jun 27/10
🧠Researchers introduce ThinkSwitch, a method that distills reasoning capabilities from large language models into smaller, more efficient models using LoRA and weight interpolation. The technique improves performance on mathematical and scientific reasoning tasks while maintaining low computational costs, doubling accuracy on AIME problems at minimal expense.
AIBullisharXiv – CS AI · May 277/10
🧠Researchers propose Early Stopping Rollout (ESR), a novel distillation technique that improves on-policy student model training by limiting rollout generation to initial response tokens. The method addresses "Off-policy Teacher Decay," where teachers lose effectiveness on later tokens, achieving better performance with higher GPU efficiency than standard approaches.
AIBullisharXiv – CS AI · May 117/10
🧠Researchers introduce ROPD, a rubric-based on-policy distillation framework that replaces teacher logits with structured semantic rubrics for model alignment. The approach achieves up to 10x better sample efficiency than logit-based methods while enabling distillation from proprietary black-box LLMs, addressing a critical scalability limitation in current model training.
AIBullisharXiv – CS AI · May 117/10
🧠Researchers introduce SOD (Step-wise On-policy Distillation), a framework that improves small language models' ability to use tools and reason through complex tasks by adaptively controlling how much they learn from larger teacher models at each step. The approach achieves up to 20.86% improvement over existing methods and demonstrates that a 0.6B parameter model can reach 26.13% accuracy on AIME 2025, a significant benchmark for mathematical reasoning.
AIBullisharXiv – CS AI · Mar 267/10
🧠Researchers introduce Hybrid Distillation Policy Optimization (HDPO), a new method that improves large language model training for mathematical reasoning by addressing 'cliff prompts' where standard reinforcement learning fails. The technique uses privileged self-distillation to provide learning signals for previously unsolvable problems, showing measurable improvements in coverage metrics while maintaining accuracy.
AIBullisharXiv – CS AI · Mar 177/10
🧠Researchers introduce MARVAL, a distillation framework that accelerates masked auto-regressive diffusion models by compressing inference into a single step while enabling practical reinforcement learning applications. The method achieves 30x speedup on ImageNet with comparable quality, making RL post-training feasible for the first time with these models.
AINeutralarXiv – CS AI · 3d ago6/10
🧠Researchers introduce Anchored Residual On-Policy Distillation (AR-OPD), a new framework for training smaller language models that improves upon existing privileged distillation methods by separating locally reachable reasoning from oracle guidance. The approach achieves 2.3-point gains over full privileged distillation and 7.9-point gains over standard supervised fine-tuning, with significant improvements on long-horizon reasoning tasks.
AINeutralarXiv – CS AI · 4d ago6/10
🧠Researchers propose Trajectory-Refined Distillation (TRD), a novel training method that addresses structural failures in on-policy distillation for large language models by correcting problematic rollouts at the trajectory level rather than token level. TRD demonstrates consistent improvements across benchmarks by mitigating prefix failure and exposing models to alternative valid reasoning paths during training.
AINeutralarXiv – CS AI · Jun 56/10
🧠Researchers propose LoRi, a low-rank distillation framework that improves implicit chain-of-thought reasoning in large language models by aligning teacher-student model trajectories in a shared low-rank tensor subspace. The method addresses the performance gap between implicit and explicit reasoning approaches, showing consistent improvements across LLaMA and Qwen model families on mathematical benchmarks.
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers propose Visual-Noise Guided In-Context Distillation (VGID), a novel framework for removing sensitive knowledge from multimodal large language models without full retraining. The method combines visual perturbation with textual in-context unlearning to achieve parameter-level knowledge removal while maintaining model performance, addressing critical privacy and safety concerns in MLLMs.
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers propose Trajectory-aware On-Policy Distillation (TOPD), a method that improves large language model reasoning by using near-future trajectory information to identify genuine reasoning divergences rather than surface-level token mismatches. The technique achieves significant performance gains on mathematical reasoning benchmarks, improving AIME24 scores from 60.0% to 63.3%.
AIBullisharXiv – CS AI · Jun 26/10
🧠Researchers present a compression pipeline for large video diffusion models that combines few-step distillation with low-bit quantization, enabling efficient deployment without sacrificing visual quality. The approach treats dual-expert denoising branches separately and achieves better results than the original model at inference speeds of 8-20 steps.
AINeutralarXiv – CS AI · Jun 16/10
🧠LARK introduces a learnability-grounded approach to trajectory selection for reasoning distillation, enabling student models to learn more efficiently from teacher-generated reasoning paths. The method uses a learnability factor to identify trajectories that maximize learning speed while maintaining distributional coverage, outperforming existing heuristic-based selection methods across multiple reasoning tasks.
AIBullisharXiv – CS AI · May 126/10
🧠Researchers introduce CA-DSSL, a new self-supervised learning technique that enables efficient AI model training on microcontrollers with under 500K parameters. The method surpasses existing approaches by 18 percentage points on standard benchmarks while requiring significantly fewer parameters, achieving 94% of supervised learning performance with models deployable in just 378 KB of memory.
AINeutralarXiv – CS AI · May 76/10
🧠Researchers introduce Budgeted LoRA, a distillation framework that compresses large language models by treating model compression as a structured compute allocation problem. The method achieves up to 4.05x speedup in inference through selective dense component removal and adaptive low-rank allocation, controlled by a single compute budget parameter.
🏢 Perplexity
AIBullisharXiv – CS AI · Apr 76/10
🧠Researchers have developed DP-OPD (Differentially Private On-Policy Distillation), a new framework for training privacy-preserving language models that significantly improves performance over existing methods. The approach simplifies the training pipeline by eliminating the need for DP teacher training and offline synthetic text generation while maintaining strong privacy guarantees.
🏢 Perplexity
AIBullisharXiv – CS AI · Mar 276/10
🧠Researchers propose X-OPD, a Cross-Modal On-Policy Distillation framework to improve Speech Large Language Models by aligning them with text-based counterparts. The method uses token-level feedback from teacher models to bridge performance gaps in end-to-end speech systems while preserving inherent capabilities.
AIBullisharXiv – CS AI · Mar 27/1024
🧠Researchers propose DUET, a new distillation-based method for LLM unlearning that removes undesirable knowledge from AI models without full retraining. The technique combines computational efficiency with security advantages, achieving better performance in both knowledge removal and utility preservation while being significantly more data-efficient than existing methods.
AIBullisharXiv – CS AI · Mar 27/1022
🧠Researchers introduce EAGLE, a reinforcement learning framework that creates unified control policies for multiple different humanoid robots without per-robot tuning. The system uses iterative generalist-specialist distillation to enable a single AI controller to manage diverse humanoid embodiments and support complex behaviors beyond basic walking.
AINeutralarXiv – CS AI · Mar 34/103
🧠DistillKac introduces a new fast image generation method using damped wave equations and Kac representation for finite-speed probability transport. Unlike diffusion models with potentially unstable reverse-time velocities, this approach enforces bounded kinetic energy and offers improved numerical stability with fewer function evaluations.
AINeutralLil'Log (Lilian Weng) · Jan 105/10
🧠Large transformer models face significant inference optimization challenges due to high computational costs and memory requirements. The article discusses technical factors contributing to inference bottlenecks that limit real-world deployment at scale.
