Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios

Researchers introduce CLI-Tool-Bench, a new benchmark for evaluating large language models' ability to generate complete software from scratch. Testing seven state-of-the-art LLMs reveals that top models achieve under 43% success rates, exposing significant limitations in current AI-driven 0-to-1 software generation despite increased computational investment.

The emergence of intent-driven development represents a fundamental shift in how software might be created, with LLMs potentially automating the entire pipeline from specification to deployment. CLI-Tool-Bench addresses a critical gap in AI evaluation methodology by moving beyond unit testing frameworks that measure isolated code snippets toward holistic end-to-end behavioral validation. This distinction matters because real-world software success depends on system-level correctness—file operations, exit codes, and terminal outputs—not just syntactic validity or local function behavior.

The benchmark's findings carry important implications for the artificial intelligence development community. Success rates below 43% among leading models indicate that despite massive investments in LLM scaling and fine-tuning, autonomous code generation at production quality remains elusive. The observation that higher token consumption fails to correlate with better performance challenges the prevailing assumption that bigger models automatically solve harder problems. The tendency toward monolithic code generation suggests LLMs struggle with architectural decomposition and modular design principles that experienced developers take for granted.

For software development organizations, these results suggest that fully autonomous code generation workflows remain premature for critical systems. The benchmark provides a rigorous foundation for tracking progress, enabling researchers to measure improvements systematically. Development teams evaluating AI-assisted tooling should recognize that current LLMs excel at specific tasks within human-guided workflows rather than genuine autonomous generation. Future work likely requires hybrid approaches combining LLM strengths with classical program synthesis techniques or architectural frameworks that guide code structure generation.

• →Top LLMs achieve under 43% success on end-to-end CLI tool generation, revealing significant gaps in autonomous software creation
• →CLI-Tool-Bench introduces black-box differential testing that validates system behavior rather than isolated code units, setting higher evaluation standards
• →Increased token consumption does not correlate with better 0-to-1 generation performance, questioning scaling assumptions
• →LLMs tend to produce monolithic code structures, indicating difficulty with architectural decomposition and modular design
• →Real-world autonomous code generation remains impractical for production systems despite recent LLM advances