Skill-Conditioned Gated Self-Distillation for LLM Reasoning

Researchers propose Skill-Conditioned Gated Self-Distillation (SGSD), a novel method for improving large language model reasoning by leveraging an experience-derived skill bank rather than trusted reference answers. The approach validates skills through a multi-teacher framework and demonstrates consistent improvements over existing methods on mathematical reasoning benchmarks.

SGSD addresses a fundamental challenge in LLM self-distillation: the reliance on trusted privileged information like correct answers or successful solution traces. This new framework shifts the paradigm by deriving supervision signals from a skill bank—compact, reusable components that may contain irrelevant or misleading information. The method's innovation lies in treating skill-based supervision as teacher hypothesis validation rather than simple imitation learning, creating a multi-teacher architecture that scores the same student output across different skill conditions.

The technical approach reflects broader trends in machine learning toward more robust and adaptive training methods. Rather than treating all supervision equally, SGSD employs a gated objective that distinguishes between informative disagreements and uncertain signals. A verifier determines whether each teacher's prediction supports success or suppresses failure, assigning positive supervision accordingly while reversing counterproductive signals. This polarity-based validation mechanism makes the system resilient to noisy or misleading skills.

Empirical results demonstrate meaningful improvements on mathematical reasoning benchmarks. On Qwen3-1.7B, SGSD outperforms GRPO by 6.2% and matches answer-conditioned OPSD by only 1.7% deficit while operating under significantly weaker assumptions about privileged information availability. These results matter because they suggest practical training improvements without requiring expensive oracle supervision, reducing the data collection burden for LLM developers.

The work positions skill-based learning as a viable alternative to reference-answer conditioning, potentially enabling more scalable and efficient LLM training pipelines. Developers working on reasoning-intensive tasks may benefit from adopting similar skill bank architectures, though the method's complexity requires careful implementation.

• →SGSD enables LLM reasoning improvement using unreliable skill banks instead of trusted reference answers, expanding accessibility of self-distillation techniques
• →Multi-teacher validation with polarity-based gating provides robustness against misleading or irrelevant skills in the supervision signal
• →Benchmark results show 6.2% improvement over GRPO and near-parity with stronger baselines while assuming weaker privileged information
• →The method demonstrates practical value for scaling LLM training without expensive oracle supervision or reference answer collection
• →Code availability enables broader adoption and reproducibility in the research and developer communities