Researchers benchmarked AI-powered peer review systems across multiple models and datasets, finding that the best configurations achieve 83% accuracy in ranking papers by quality and catch 71.6% of intentionally injected errors. While AI review systems show promise in tracking human quality judgments and earning positive user feedback, they still require substantial improvement before serving as primary peer review mechanisms.
The emergence of agentic review systems represents a significant response to mounting pressure on traditional peer review infrastructure as AI-assisted research accelerates publication volumes. This study systematically evaluates how well AI review systems can replicate human scholarly judgment—a critical question for the research community's ability to maintain quality standards at scale. The benchmarking reveals a clear capability gradient: OpenAIReview with GPT-5.5 outperforms other configurations, suggesting that system design and underlying model choice substantially impact performance. However, the 71.6% error detection rate demonstrates meaningful gaps remain. The finding that different models detect different error categories suggests that ensemble approaches combining multiple AI systems could improve outcomes, pointing toward hybrid architectures rather than single-system deployments. Real-world deployment data showing positive vote ratios (1.44:1) indicates users find value despite imperfections, though complaints about false positives highlight remaining calibration challenges. The research positions AI review as a complementary tool rather than replacement technology. For the AI research community, this work establishes methodology for continuous improvement of review systems and identifies specific error categories requiring targeted enhancement. The 83.3% recall achieved through model ensemble suggests ceiling performance may exceed individual system capabilities, influencing future development priorities. This benchmark framework enables ongoing evaluation as frontier models improve, making it a foundational reference for assessing AI review system maturity.
- →OpenAIReview + GPT-5.5 achieves 83% pairwise accuracy in ranking papers by quality against citation and acceptance signals.
- →Current best AI review system detects only 71.6% of injected errors, indicating substantial room for technical improvement.
- →Ensemble approaches combining six models reach 83.3% error detection recall, suggesting different models capture distinct error types.
- →Real-world deployment shows positive user sentiment (1.44:1 positive-to-negative vote ratio) but reveals false positives as primary complaint.
- →Study establishes comprehensive benchmarking methodology for evaluating agentic review systems against ground truth and external quality signals.