SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents

Researchers introduce SkillRet, a large-scale benchmark dataset containing 17,810 public agent skills designed to evaluate how language model agents retrieve appropriate tools from massive skill libraries. The benchmark demonstrates that current retrieval methods struggle significantly with realistic large-scale deployments, though task-specific fine-tuning improves performance by up to 16.9 points on key metrics.

SkillRet addresses a critical infrastructure gap in the rapidly advancing field of agentic AI systems. As language model agents become more sophisticated and capable, they're increasingly deployed with access to extensive libraries of reusable skills and tools. The challenge of efficiently selecting the correct skill for a given user request has grown from a theoretical problem to a practical bottleneck affecting system performance and cost. This benchmark tackles an underexplored area where existing solutions prove inadequate at scale.

The emergence of large-scale agent frameworks has revealed that explicit skill naming—viable in small ecosystems—becomes impractical when libraries contain tens of thousands of tools. Context windows and latency budgets create hard constraints that make naive approaches ineffective. SkillRet's contribution lies in providing both a standardized evaluation framework and training data that enables researchers to develop better retrieval mechanisms. The benchmark's organization with semantic tags and hierarchical categorization mirrors real-world skill ecosystems.

For the AI development community, this benchmark serves as a foundation for improving agent reliability and efficiency. The substantial performance gaps between fine-tuned and off-the-shelf models indicate significant optimization potential remains. Better skill retrieval directly impacts user experience, system latency, and cost efficiency in production agent systems. Developers building agent platforms will benefit from refined retrieval techniques derived from SkillRet research.

Looking forward, this benchmark will likely accelerate research into neural retrieval methods optimized for agent workflows. Future work may explore cross-domain skill transfer, dynamic skill library updates, and hybrid retrieval-reasoning approaches that combine semantic matching with task-specific reasoning.

• →SkillRet benchmark contains 17,810 public skills with 63,259 training samples, establishing the first large-scale evaluation framework for agent skill retrieval.
• →Off-the-shelf retrieval models significantly underperform on realistic skill libraries, revealing skill selection as a critical unsolved challenge in agent deployment.
• →Task-specific fine-tuning improves NDCG@10 by 16.9 points over baseline retrievers, demonstrating substantial optimization potential.
• →Fine-tuned models succeed by focusing on relevant signals within long, noisy queries rather than processing entire request texts equally.
• →The benchmark addresses a practical systems bottleneck affecting performance, latency, and cost efficiency of production agent systems.