MENTOR: Reinforcement Learning via Flexible Teacher-Optimized Rewards for Tool-Use Distillation

Researchers propose MENTOR, a reinforcement learning framework that improves how small language models learn tool-use capabilities from larger models by using flexible, process-aware rewards instead of rigid trajectory replication. The approach demonstrates better out-of-domain generalization than supervised fine-tuning and strict RL baselines in executable-tool environments.

MENTOR addresses a fundamental challenge in AI model distillation: efficiently transferring specialized capabilities from large language models to smaller, more deployable versions. Traditional supervised fine-tuning creates brittle models that fail when encountering tasks beyond their training distribution, while strict reinforcement learning approaches either starve smaller models of meaningful guidance or force them to perfectly replicate teacher behavior despite their limited capacity.

The research builds on growing recognition that model distillation requires fundamentally different approaches than standard training. As enterprises increasingly deploy edge-based AI systems with computational constraints, the ability to compress sophisticated tool-use capabilities into smaller models becomes strategically valuable. Previous work established that trajectory matching alone produces poor generalization, yet outcome-only rewards fail to adequately guide smaller models through complex decision sequences.

MENTOR's flexible reward structure occupies a practical middle ground by using teacher trajectories as behavioral guidance rather than strict targets. This nuanced approach acknowledges that smaller models may discover alternative valid paths to achieve tool-use objectives. The framework's effectiveness specifically in verifiable tool environments—where task execution can be validated objectively—suggests practical deployment pathways for autonomous agent systems that require both capability and reliability.

The implications extend beyond academic machine learning. As AI-powered applications move toward decentralized or resource-constrained deployment scenarios, efficient model distillation becomes infrastructure-critical. The findings suggest that future AI systems may benefit from process-aware training approaches rather than end-to-end imitation, potentially enabling more robust autonomous agents at reduced computational cost.

• →MENTOR's flexible reward structure outperforms rigid trajectory matching for teaching tool-use to smaller language models
• →Out-of-domain generalization improves significantly when models have behavioral guidance without strict replication requirements
• →The approach demonstrates that model capacity constraints require fundamentally different distillation strategies than larger model training
• →Process-aware rewards offer more effective learning than outcome-only or trajectory-matching baselines in verifiable environments
• →Practical deployment of capable smaller models requires balancing behavioral alignment with performance optimization