17 articles tagged with #model-distillation. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
AIBullisharXiv – CS AI · Apr 107/10
🧠Researchers developed a weak supervision framework to detect hallucinations in large language models by distilling grounding signals into transformer representations during training. Using substring matching, sentence embeddings, and LLM judges, they created a 15,000-sample dataset and trained five probing classifiers that achieve hallucination detection from internal activations alone at inference time, eliminating the need for external verification systems.
AIBullisharXiv – CS AI · Mar 117/10
🧠Researchers have developed Meissa, a lightweight 4B-parameter medical AI model that brings advanced agentic capabilities offline for healthcare applications. The system matches frontier models like GPT in medical benchmarks while operating with 25x fewer parameters and 22x lower latency, addressing privacy and cost concerns in clinical settings.
🧠 Gemini
AIBullisharXiv – CS AI · Mar 57/10
🧠Researchers propose LEAP, a new framework for detecting AI hallucinations using efficient small models that can dynamically adapt verification strategies. The system uses a teacher-student approach where a powerful model trains smaller ones to detect false outputs, addressing a critical barrier to safe AI deployment in production environments.
AIBullisharXiv – CS AI · Mar 46/105
🧠Researchers developed a three-stage curriculum learning framework that improves Chain-of-Thought reasoning distillation from large language models to smaller ones. The method enables Qwen2.5-3B-Base to achieve 11.29% accuracy improvement while reducing output length by 27.4% through progressive skill acquisition and Group Relative Policy Optimization.
AIBullisharXiv – CS AI · Feb 277/106
🧠Researchers developed ViT-Linearizer, a distillation framework that transfers Vision Transformer knowledge into linear-time models, addressing quadratic complexity issues for high-resolution inputs. The method achieves 84.3% ImageNet accuracy while providing significant speedups, bridging the gap between efficient RNN-based architectures and transformer performance.
AIBullisharXiv – CS AI · Feb 277/108
🧠Researchers propose Generalized On-Policy Distillation (G-OPD), a new AI training framework that improves upon standard on-policy distillation by introducing flexible reference models and reward scaling factors. The method, particularly ExOPD with reward extrapolation, enables smaller student models to surpass their teacher's performance in math reasoning and code generation tasks.
AIBullisharXiv – CS AI · 3d ago6/10
🧠Researchers introduce HintMR, a hint-assisted reasoning framework that improves mathematical problem-solving in small language models by using a separate hint-generating model to provide contextual guidance through multi-step problems. This collaborative two-model system demonstrates significant accuracy improvements over standard prompting while maintaining computational efficiency.
AINeutralarXiv – CS AI · 3d ago6/10
🧠Researchers attempted to train behavioral dispositions into small language models through distillation but found that initial positive results were artifacts of measurement errors. After rigorous validation, they discovered no reliable method to instill self-verification and uncertainty acknowledgment without degrading model performance or creating superficial stylistic mimicry across five different small models.
AINeutralarXiv – CS AI · 3d ago6/10
🧠Researchers investigate on-policy distillation (OPD) dynamics in large language model training, identifying two critical success conditions: compatible thinking patterns between student and teacher models, and genuine new capabilities from the teacher. The study reveals that successful OPD relies on token-level alignment and proposes recovery strategies for failing distillation scenarios.
AIBullisharXiv – CS AI · Apr 76/10
🧠Researchers developed a new training approach that makes small language models more effective search agents by teaching them to consistently use search tools rather than relying on internal knowledge. The method achieved significant performance improvements of 17.3 points on Bamboogle and 15.3 points on HotpotQA, reaching large language model-level results while maintaining lower computational costs.
AIBullisharXiv – CS AI · Mar 266/10
🧠Researchers introduce Uni-DAD, a unified approach that combines diffusion model distillation and adaptation into a single pipeline for efficient few-shot image generation. The method achieves comparable quality to state-of-the-art methods while requiring less than 4 sampling steps, addressing the computational cost issues of traditional diffusion models.
AIBullisharXiv – CS AI · Mar 176/10
🧠Researchers developed a framework to make large language model-based query expansion more efficient by distilling knowledge from powerful teacher models into compact student models. The approach uses retrieval feedback and preference alignment to maintain 97% of the original performance while dramatically reducing inference costs.
AIBullisharXiv – CS AI · Mar 126/10
🧠Researchers introduce HEAL (Hindsight Entropy-Assisted Learning), a new framework for distilling reasoning capabilities from large AI models into smaller ones. The method overcomes traditional limitations by using three core modules to bridge reasoning gaps and significantly outperforms standard distillation techniques.
🏢 Perplexity
AIBullisharXiv – CS AI · Mar 96/10
🧠Researchers introduce TempoSyncDiff, a new AI framework that uses distilled diffusion models to generate realistic talking head videos from audio with significantly reduced computational latency. The system addresses key challenges in AI-driven video synthesis including temporal instability, identity drift, and audio-visual alignment while enabling deployment on edge computing devices.
AIBullisharXiv – CS AI · Feb 276/107
🧠Researchers propose Struct-SQL, a knowledge distillation framework that improves Small Language Models for Text-to-SQL tasks by using structured Chain-of-Thought reasoning instead of unstructured approaches. The method achieves an 8.1% improvement over baseline distillation, primarily by reducing syntactic errors through formal query execution plan blueprints.
AIBullishHugging Face Blog · Nov 196/106
🧠The article discusses Apriel-H1, a methodology or framework for creating more efficient reasoning models in AI. This approach appears to focus on distillation techniques to improve model performance while reducing computational requirements.
AIBullishOpenAI News · Oct 16/106
🧠OpenAI introduces model distillation capabilities in their API, allowing developers to fine-tune smaller, cost-efficient models using outputs from larger frontier models. This feature enables users to create optimized models that balance performance and cost within OpenAI's platform ecosystem.
