ToolSense: A Diagnostic Framework for Auditing Parametric Tool Knowledge in LLMs

Researchers introduce ToolSense, a diagnostic framework that reveals significant gaps in how large language models understand tools despite strong retrieval performance. Testing on ~47k tools shows parametric models collapse by 50-64% on realistic queries compared to benchmark performance, suggesting current evaluation methods mask fundamental knowledge deficiencies.

ToolSense addresses a critical blind spot in LLM agent evaluation. While parametric tool retrieval—encoding tools as virtual tokens within model vocabularies—has dominated recent benchmarks, these evaluations rely on verbose, fully-specified queries with constrained decoding that artificially inflate performance metrics. The framework's findings expose a troubling knowledge-retrieval dissociation: models achieving strong scores on standard ToolBench evaluations collapse dramatically when facing realistic, ambiguous queries, sometimes falling below simpler embedding-based baselines.

This research emerges from the broader challenge of scaling LLM agents across massive tool catalogs. As enterprises deploy agents with thousands of available tools, both embedding-based and parametric retrieval approaches compete for dominance. The parametric approach's theoretical advantage—using the full LLM's contextual understanding rather than compact encoders—appeared validated by benchmark results. However, ToolSense reveals these benchmarks don't test genuine tool comprehension, only memorization patterns within constrained evaluation conditions.

For the AI development community, this finding carries immediate implications. Teams optimizing models on existing ToolBench metrics may unknowingly building systems that perform poorly in production environments with real user queries. The framework's three-tier ambiguity structure and factual probing benchmarks enable more rigorous evaluation of tool understanding beyond statistical pattern matching. The near-random performance on factual knowledge probes despite strong retrieval scores suggests models learn surface-level token associations rather than semantic tool understanding.

Moving forward, the research signals a shift toward diagnostic evaluation frameworks that stress-test agent capabilities under realistic conditions. The open-sourced ToolSense provides developers tools to audit their own implementations, potentially driving broader adoption of more rigorous evaluation standards across the LLM agent ecosystem.

• →Parametric tool retrieval models show 50-64% performance collapse on realistic queries despite strong ToolBench benchmark results
• →Knowledge-retrieval dissociation reveals models achieving high retrieval scores score near-random on factual tool understanding probes
• →Standard ToolBench evaluation methodology masks model deficiencies through verbose query specifications and constrained decoding
• →ToolSense framework generates three diagnostic benchmarks with varying ambiguity tiers to reveal genuine tool comprehension gaps
• →Open-sourced tools enable developers to audit their LLM agent implementations against more realistic evaluation standards