SOD: Step-wise On-policy Distillation for Small Language Model Agents

Researchers introduce SOD (Step-wise On-policy Distillation), a framework that improves small language models' ability to use tools and reason through complex tasks by adaptively controlling how much they learn from larger teacher models at each step. The approach achieves up to 20.86% improvement over existing methods and demonstrates that a 0.6B parameter model can reach 26.13% accuracy on AIME 2025, a significant benchmark for mathematical reasoning.

The research addresses a fundamental challenge in deploying AI agents on resource-constrained devices: smaller language models struggle with tool-use reasoning because they make mistakes that compound across multiple reasoning steps. Traditional on-policy distillation methods, which have proven effective in other contexts, fail here because erroneous tool calls create divergence between student and teacher outputs, causing the teacher's guidance to become unreliable and misleading.

The breakthrough lies in SOD's adaptive reweighting mechanism. Rather than applying uniform distillation strength throughout a reasoning trajectory, the framework dynamically adjusts supervision intensity based on step-level divergence. This prevents the model from over-learning from corrupted teacher signals while maintaining dense guidance in well-aligned regions. The approach bridges the gap between sparse reinforcement learning rewards and potentially harmful dense supervision.

For the AI development community, this work has immediate practical implications. Deploying reasoning-capable agents on mobile devices, edge hardware, and low-bandwidth environments becomes more feasible when 0.6B models can achieve competitive performance on challenging benchmarks. The 26.13% AIME score from a sub-billion parameter model represents meaningful progress toward accessible agentic AI.

The technical contribution matters beyond mathematics and science tasks—the framework generalizes across math, science, and code domains, suggesting broad applicability. Organizations developing lightweight AI systems for production environments can adopt this distillation strategy to improve reasoning reliability without substantial computational overhead, potentially accelerating the deployment of tool-integrated AI in resource-limited settings.

• →SOD addresses cascading error failures in small model tool-use by adaptively controlling distillation strength per reasoning step
• →A 0.6B parameter student model achieves 26.13% on AIME 2025, demonstrating effective transfer of complex reasoning to lightweight models
• →The framework improves performance up to 20.86% over existing baselines across math, science, and code benchmarks
• →Adaptive reweighting prevents misleading teacher signals in high-divergence regions while preserving supervision in aligned states
• →The approach enables deployment of reasoning-capable agents on resource-constrained devices without substantial performance degradation