AssertLLM2: A Comprehensive LLM Benchmark for Assertion Generation from Design Specifications

Researchers introduce AssertLLM2, an open-source benchmark containing 83 real-world hardware designs to evaluate how well Large Language Models can automatically generate formal SystemVerilog Assertions from specifications. The benchmark uniquely incorporates buggy RTL variants to assess both bug prevention and bug detection capabilities, establishing more rigorous evaluation standards for LLM-assisted hardware verification.

AssertLLM2 addresses a critical gap in hardware verification automation by providing the first comprehensive benchmark that realistically evaluates LLMs on assertion generation tasks. Assertion-based verification remains essential for catching design flaws before manufacturing, yet the manual process of translating design intent into formal assertions consumes significant engineering resources and introduces human error. This benchmark directly tackles that bottleneck by offering 83 real-world designs across 13 functional categories with structured specifications, verified golden RTL implementations, and systematically mutated buggy variants.

The significance lies in how AssertLLM2 moves beyond previous academic benchmarks that relied on simplified specifications and unrealistic task formulations. By including buggy RTL as explicit input, the benchmark enables evaluation of LLM capabilities in practical bug-hunting scenarios—assessing whether generated assertions can actually detect implementation errors. This mirrors real-world verification workflows where engineers iteratively refine assertions to catch subtle design flaws.

For the hardware design industry, this benchmark establishes a standardized evaluation framework that goes beyond syntactic validity to measure formal provability, coverage metrics, and mutation-based bug detection rates. This enables more meaningful comparisons between LLM approaches and helps identify which models and techniques are genuinely production-ready. The open-source nature democratizes access to rigorous evaluation tools, accelerating development of assertion automation technologies.

As semiconductor design complexity continues escalating, LLM-assisted verification automation could significantly reduce time-to-market and improve design quality. AssertLLM2 provides the foundation for identifying which LLM approaches genuinely advance the field versus those that merely perform well on simplified benchmarks.

• →AssertLLM2 is the first benchmark to evaluate LLMs on assertion generation using realistic hardware designs and buggy RTL variants
• →The benchmark covers 83 real-world designs across 13 functional categories with structured specifications and verified golden implementations
• →Evaluation spans syntactic validity, formal provability, coverage, and mutation-based bug detection for comprehensive assessment
• →This addresses a critical industry bottleneck where manual assertion creation consumes significant engineering resources in hardware verification
• →Open-source accessibility enables standardized evaluation of LLM approaches for practical hardware verification automation