Tool-Genesis: A Task-Driven Tool Creation Benchmark for Self-Evolving Language Agent

Researchers introduce Tool-Genesis, a new benchmark for evaluating self-evolving AI agents' ability to create and use tools from abstract requirements. The study reveals that even advanced AI models struggle with creating precise tool interfaces and executable logic, with small initial errors causing significant downstream performance degradation.

• →Tool-Genesis benchmark evaluates AI agents across interface compliance, functional correctness, and downstream utility dimensions.
• →Existing benchmarks rely too heavily on predefined specifications, limiting autonomous AI evolution assessment.
• →State-of-the-art AI models struggle to produce accurate tool interfaces or executable logic in one-shot scenarios.
• →Minor initial flaws in tool creation amplify through the pipeline, causing sharp performance degradation.
• →The benchmark aims to guide future research toward training models that synthesize persistent, general-purpose tools.