DeepTool: Scaling Interleaved Deliberation in Tool-Integrated Reasoning via Process-Supervised Reinforcement Learning

DeepTool is a new AI framework that enhances large language models' ability to reason through tool use by implementing process-supervised reinforcement learning. The system dramatically improves performance on mathematical benchmarks like AIME24 (3.2% to 40.4%) while maintaining token efficiency through interleaved thinking and action.

DeepTool represents a meaningful advance in bridging the gap between LLM reasoning capabilities and practical tool execution. Traditional approaches to tool-integrated reasoning suffer from sparse reward signals that only evaluate final outcomes, leaving intermediate reasoning steps unsupervised and prone to error accumulation. By introducing process supervision through an Action-Centric Process Reward mechanism, DeepTool guides models through each deliberative cycle of thinking, acting, and observing—fundamentally changing how models approach sequential problem-solving.

This work emerges from broader trends in AI development where researchers recognize that raw capability alone isn't sufficient; models need structured feedback mechanisms during execution to develop robust planning and self-correction behaviors. The synthesis pipeline incorporating adversarial perturbations suggests the framework prioritizes reliability over raw performance metrics, addressing practical deployment concerns.

For the AI development community, these results matter substantially. A 37-point improvement on AIME24 and 28.6% performance on HMMT25—problems historically challenging for smaller models—indicates that process-level supervision can unlock capabilities previously thought to require larger model scales. This has economic implications for organizations seeking competitive performance without deploying massive parameter models. The token cost-effectiveness analysis validates that improved efficiency doesn't come from shortcuts but from better reasoning architecture.

The immediate relevance centers on whether this approach generalizes beyond mathematical reasoning to other tool-using domains like code execution, database queries, and complex planning tasks. The research trajectory suggests process supervision may become a standard technique in production AI systems, particularly where sequential decision-making and error correction are critical.

• →DeepTool uses process-supervised reinforcement learning to supervise intermediate steps in tool-integrated reasoning, not just final outcomes.
• →The framework boosts Qwen2.5-7B performance dramatically on AIME24 (3.2% to 40.4%) and HMMT25 (0% to 28.6%) benchmarks.
• →Adversarial perturbations in the synthesis pipeline enhance robustness and self-correction during tool invocation.
• →Action-Centric Process Rewards reinforce precise tool usage at every step rather than relying solely on sparse outcome-based signals.
• →The approach demonstrates optimal balance between performance gains and token efficiency, making smaller models more competitive.