Verus-SpecGym: An Agentic Environment for Evaluating Specification Autoformalization

Researchers introduce Verus-SpecGym, an evaluation environment for testing whether AI agents can automatically translate informal programming specifications into formal, machine-verifiable code. The benchmark reveals that frontier LLMs like Gemini 3.1 Pro achieve 77.8% accuracy on specification tasks, but generated specs remain brittle and frequently miss edge cases, input constraints, and validation rules that human experts catch.

Specification autoformalization represents a critical frontier in AI-assisted formal verification, addressing a gap between what code does and what users actually intend. The research tackles the fundamental problem that while AI agents can generate code with machine-checked proofs, no mechanism currently ensures the formal specification itself reflects user intent. Verus-SpecGym solves a crucial evaluation challenge by leveraging Codeforces test cases and adversarial hacks—edge cases written by competitive programmers—to validate automatically-generated specifications against real-world correctness criteria.

This work builds on growing momentum in AI-assisted formal methods. Major organizations increasingly deploy coding agents for real software, creating urgent demand for reliable verification pipelines. The Verus platform's extension to execute specifications as Rust code enables comprehensive testing that traditional LLM judging cannot match—notably missing 26% of failures that executable testing catches.

The performance gap between frontier and open-source models (77.8% vs. 21.5–25.5%) highlights how specification writing demands nuanced reasoning about implicit assumptions and boundary conditions. Frontier models still struggle with omitting input constraints and accepting invalid outputs, suggesting specification autoformalization remains fundamentally challenging despite code generation advances.

For developers and enterprises deploying AI agents in safety-critical systems, these results indicate that specification generation cannot yet operate autonomously. The brittleness of even frontier-model specifications argues for hybrid approaches combining agent assistance with human review. Future research should focus on enabling agents to identify and test specification edge cases systematically, potentially through iterative refinement against comprehensive test suites.

• →Gemini 3.1 Pro achieves 77.8% accuracy on specification autoformalization tasks, substantially outperforming other frontier models at 51–58%.
• →AI-generated specifications frequently omit critical input assumptions, accept incorrect outputs, and fail on edge cases despite agents generating correct code.
• →Executable specification testing catches 26% more failures than LLM-based evaluation, demonstrating the necessity of rigorous verification methods.
• →Open-source models achieve only 21.5–25.5% accuracy, widening the capability gap between frontier and accessible AI systems for formal verification tasks.
• →Specification autoformalization remains brittle and unsuitable for fully autonomous deployment in safety-critical systems without human oversight.