SmartEval: A Benchmark for Evaluating LLM-Generated Smart Contracts from Natural Language Specifications

Researchers introduce SmartEval, a comprehensive benchmark for evaluating Solidity smart contracts generated by LLMs from natural language specifications, comprising 9,000 contracts with expert validation and a five-dimensional evaluation framework. The study reveals characteristic failure modes in LLM-generated contracts and confirms that automated evaluation scores align closely with human expert judgment, establishing a reproducible foundation for assessing smart contract synthesis quality.

SmartEval addresses a critical gap in AI-assisted blockchain development by providing the first large-scale, systematically validated benchmark for evaluating LLM-generated smart contracts. As organizations increasingly explore using language models to accelerate smart contract development, understanding the reliability and failure patterns of these systems becomes essential. The benchmark's five-dimensional rubric—covering functional completeness, variable fidelity, state-machine correctness, business-logic fidelity, and code quality—provides granular insight into where LLMs succeed and struggle.

The empirical validation strengthens the benchmark's credibility. Human expert evaluation by Columbia University PhD researchers confirmed that automated scores align within 0.34 points of expert judgment, while Slither static analyzer agreement at 79.4% demonstrates consistency with established security tools. These validations matter because they establish that the benchmark measures meaningful quality differences, not arbitrary metrics.

The study's findings reveal important patterns: logic omissions occur in 35.3% of failures, state transition errors in 23.4%, and complexity-driven degradation affects contracts as specifications grow intricate. Counterintuitively, LLM-generated contracts scored 8.29 points higher than ground-truth implementations, attributed to their literal specification-following behavior rather than superior engineering—a nuance that developers must understand when deploying AI-assisted code.

For the blockchain ecosystem, SmartEval enables developers to measure LLM reliability before deployment and helps researchers identify specific areas for model improvement. Security-conscious teams can now benchmark their AI-assisted workflows against validated standards. The public release of all data, evaluation code, and 9,000 generated contracts democratizes this research, potentially accelerating improvements in LLM smart contract quality across the industry.

• →SmartEval's five-dimensional evaluation rubric and 9,000 contract corpus provide the first large-scale validated benchmark for assessing LLM-generated smart contracts.
• →Human expert validation confirmed automated evaluation scores align within 0.34 points of Columbia PhD researchers' assessments, establishing benchmark credibility.
• →LLM-generated contracts exhibit characteristic failure modes: logic omissions (35.3%), state transition errors (23.4%), and complexity-driven degradation as specifications increase.
• →Counterintuitively, LLM contracts scored higher than ground-truth implementations due to literal specification-following behavior, not superior code quality.
• →Public release of evaluation framework and 9,000 contracts enables developers to benchmark AI-assisted smart contract development workflows against validated standards.