CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers

Researchers introduce CoCoReviewBench, a new benchmark dataset of 3,900 papers from ICLR and NeurIPS designed to reliably evaluate AI review systems. The benchmark addresses critical gaps in current evaluation methods by prioritizing correctness over mere overlap with human reviews, revealing that existing AI reviewers struggle with hallucinations and reasoning accuracy.

The development of CoCoReviewBench tackles a fundamental problem in AI evaluation: existing metrics for assessing AI reviewers rely too heavily on matching human reviews, which themselves are often incomplete and contain errors. This creates a circular validation problem where flawed human reviews become the gold standard. The researchers addressed this by implementing category-specific subsets and filtering unreliable reviews through expert annotations derived from reviewer-author-meta-review discussions, establishing a more robust evaluation framework.

The broader context reveals the increasing importance of AI systems in academic peer review as research volume explodes. Traditional human review processes face bandwidth constraints, creating demand for AI assistance. However, deploying unreliable AI reviewers introduces quality risks that could damage scientific integrity. This benchmark directly serves the growing AI review automation market by providing a credible evaluation standard.

For the AI development community, CoCoReviewBench establishes clearer performance baselines and exposes critical weaknesses—particularly hallucinations and reasoning failures in current systems. The finding that reasoning-focused models outperform standard approaches suggests a specific technical direction for improvement. This benchmark enables more systematic progress toward production-ready AI review systems that could eventually augment human reviewers in academic workflows.

Looking ahead, the availability of this benchmark will likely accelerate research in AI reviewer systems, potentially attracting corporate investment in academic automation tools. The emphasis on correctness over coverage sets a new standard for responsible AI deployment in high-stakes domains.

• →CoCoReviewBench provides a 3,900-paper dataset that evaluates AI reviewers on correctness rather than human-review overlap, addressing fundamental evaluation gaps.
• →Current AI reviewers demonstrate significant limitations including hallucinations and weak reasoning capabilities across academic review tasks.
• →Reasoning-oriented models show measurably better performance as reviewers, indicating a promising research direction for improvement.
• →Expert annotations from reviewer-author-meta-review discussions replace flawed human reviews as reliability benchmarks.
• →The benchmark establishes credibility standards for deploying AI systems in high-stakes academic peer review environments.