VeriContest: A Competitive-Programming Benchmark for Verifiable Code Generation

Researchers introduce VeriContest, a benchmark of 946 competitive-programming problems designed to evaluate AI models' ability to generate not just functional code but also formal specifications and machine-checkable proofs. Testing ten state-of-the-art models reveals a dramatic capability gap: while the strongest model achieves 92% accuracy on code generation alone, performance plummets to 48% on specifications, 14% on proofs, and just 5% end-to-end, identifying proof generation as the critical bottleneck for verifiable code generation systems.

VeriContest addresses a fundamental gap in AI code-generation evaluation by moving beyond traditional testing paradigms toward formal verification. Current benchmarks fail to measure whether generated code includes machine-checkable correctness proofs—a critical requirement for high-stakes software development. The benchmark's construction through expert validation and semi-automated expansion ensures quality that mirrors real-world verification standards rather than academic abstractions.

The stark performance disparity reveals why verifiable code generation remains nascent despite rapid advances in general code synthesis. Models excel at pattern matching for functional correctness but struggle with the abstract reasoning required for formal specification writing and proof construction. This gap explains why LLM-generated code still requires extensive human review despite impressive benchmarks in coding competitions.

For the software development industry, VeriContest establishes rigorous measurement criteria for tools that could eventually reduce critical vulnerabilities in deployed systems. The benchmark's focus on Rust and Verus positions it within the growing push toward memory-safe languages and formal methods adoption across infrastructure projects.

Looking forward, the benchmark will likely drive model development toward hybrid approaches combining code generation with automated theorem-proving assistance. Organizations building verification-critical systems should monitor progress on these specific metrics—specification and proof generation—as indicators of when AI-assisted formal verification becomes practically viable for production environments.

• →VeriContest benchmarks 946 problems to evaluate verifiable code generation including specifications and proofs, not just functional code.
• →Top models achieve 92% on code generation but only 5% end-to-end on verified synthesis, revealing proof generation as the bottleneck.
• →The benchmark uses expert validation and testing as quality assurance, ensuring results reflect real-world verification standards.
• →Results demonstrate models lack abstract reasoning capabilities needed for formal specification and proof writing despite strong pattern-matching abilities.
• →The benchmark establishes measurable criteria for evaluating AI progress toward trustworthy software generation for high-stakes applications.