RTL-BenchLS: A Large-Scale Benchmark for RTL Reasoning and Generation with Large Language Models

Researchers introduce RTL-BenchLS, a large-scale benchmark containing over 10,000 formally verified Verilog designs for evaluating large language models on hardware design tasks. The benchmark addresses limitations of existing datasets through three novel self-supervised tasks beyond specification-to-RTL generation, with top models achieving only 12-28% accuracy, demonstrating substantial room for improvement in LLM-based hardware automation.

RTL-BenchLS represents a significant infrastructure advancement for hardware design automation. Existing benchmarks have become saturated by frontier LLMs due to limited scale and scope, with designs that are relatively simple and tasks narrowly focused on specification-to-RTL conversion. This new benchmark fundamentally addresses the data scarcity problem by introducing self-supervised tasks that eliminate the need for manually aligned labels, directly solving the bottleneck that has constrained benchmark expansion.

The research emerges from a broader trend toward automating hardware design through AI. As semiconductor complexity increases and design cycles compress, LLM-assisted RTL generation has gained traction as a potential productivity multiplier. However, progress measurement has been hampered by inadequate evaluation frameworks. RTL-BenchLS contains over 10,000 formally verified designs—orders of magnitude larger than predecessors—and introduces three distinct evaluation tasks: round-trip reasoning (paraphrasing specifications), masked-content reasoning (inferring missing code), and repository-issue reasoning (fixing real-world design bugs).

For the AI and hardware industries, this benchmark provides critical calibration on LLM capabilities. The results revealing only 12-28% accuracy on novel tasks indicate that LLMs still lack robust reasoning for complex hardware design, despite their general intelligence. This gap suggests immediate opportunities for specialized model development and fine-tuning approaches targeting hardware domains. The formal equivalence verification method eliminates manual testing overhead, enabling future benchmark expansion as LLM-generated designs improve.

Developers and researchers should anticipate this becoming a standard evaluation framework. The benchmark's rigor and scale will likely establish baseline expectations for LLM performance in hardware automation, potentially accelerating development of specialized tools and methodologies for RTL generation.

• →RTL-BenchLS contains 10,000+ formally verified Verilog designs, substantially larger than existing hardware design benchmarks.
• →Three novel self-supervised tasks eliminate manual label creation bottleneck, enabling benchmark scalability.
• →Top LLM performance ranges from 12-28% accuracy, revealing substantial gaps in AI hardware reasoning capabilities.
• →Formal equivalence checking replaces manual testbenches, establishing rigorous verification standards for benchmark evaluation.
• →The benchmark signals that current frontier models lack sufficient capability for production-level hardware design automation.