Disambiguation-Centric Finetuning Makes Enterprise Tool-Calling LLMs More Realistic and Less Risky

Researchers introduce DiaFORGE, a three-stage framework for training LLMs to reliably invoke enterprise APIs by focusing on disambiguation between similar tools and underspecified arguments. Fine-tuned models achieved 27-49 percentage points higher tool-invocation success than GPT-4o and Claude-3.5-Sonnet, with an open corpus of 5,000 production-grade API specifications released for further research.

Enterprise adoption of LLMs as autonomous agents faces a critical reliability bottleneck: models frequently misidentify which tool to call when presented with similar options or incomplete user requests. DiaFORGE addresses this gap through a practical engineering solution that treats disambiguation as a first-class training objective rather than an afterthought. The framework synthesizes realistic multi-turn dialogues where clarification is necessary, fine-tunes open-source models with explicit reasoning traces, and validates performance in live agentic loops rather than static benchmarks—a methodological shift that mirrors real-world deployment constraints.

This work reflects a maturing understanding of LLM limitations in production environments. While GPT-4o and Claude perform well on benchmark prompting, they fail dramatically on edge cases that demand nuanced tool selection. The 27-49 percentage point improvements signal that targeted, domain-aware fine-tuning remains a high-leverage lever for reliability, even as foundation models grow larger. The release of 5,000 curated API specifications with validation represents significant infrastructure investment in standardizing agent training data.

For enterprises evaluating LLM-based automation, this research validates the viability of open-source alternatives fine-tuned on disambiguation-focused data over relying solely on closed APIs. The emphasis on dynamic evaluation—deploying models in live loops—sets a new bar for what "production-ready" means in agentic AI. Organizations building internal tool-calling systems should expect similar dramatic gains from disambiguation-centric training, making this both a technical contribution and a practical blueprint for reducing deployment risk.

• →DiaFORGE-trained models outperformed GPT-4o by 27 pp and Claude-3.5-Sonnet by 49 pp on tool-invocation tasks involving similar options or underspecified arguments.
• →Disambiguation-focused fine-tuning on 3B-70B parameter models proves more effective than prompting closed APIs for enterprise tool-calling reliability.
• →An open corpus of 5,000 production-grade API specifications with validated disambiguation dialogues is now available, lowering barriers to building robust agentic systems.
• →Dynamic evaluation in live agentic loops revealed performance gaps missed by static benchmarks, establishing a new evaluation standard for production-ready agents.
• →Open-source models fine-tuned on targeted, domain-specific data can match or exceed closed model performance on specialized agent tasks.