PeerCheck: Enhancing LLM-Generated Academic Reviews Towards Human-Level Quality

Researchers introduce PeerCheck, a framework that analyzes differences between LLM-generated and human-written academic reviews, finding that LLMs prioritize theoretical aspects while humans emphasize methodology. Using techniques like Chain-of-Thought prompting improves LLM review quality, though retrieval-augmented generation surprisingly produces inconsistent and sometimes degraded results.

The academic peer review system faces mounting pressure as submission volumes exceed human capacity to maintain quality standards. This research addresses a practical problem: as institutions increasingly consider LLM assistance for review tasks, understanding how AI-generated reviews differ from human evaluations becomes critical for maintaining scientific integrity. The PeerCheck framework provides empirical evidence that LLMs and humans have fundamentally different evaluation priorities, with machines focusing on theoretical soundness while humans scrutinize experimental rigor and methodological choices.

The findings emerge from a broader trend of AI integration into knowledge work. Universities and publishers are exploring LLMs to accelerate review processes, but without proper validation, this could degrade paper quality and introduce systematic biases. The research's discovery that Chain-of-Thought prompting significantly enhances LLM review quality offers a practical improvement pathway for institutions implementing AI-assisted review. However, the unexpected "RAG paradox"—where augmenting LLMs with external information sometimes worsens performance—reveals non-intuitive limitations in current AI capabilities.

For the academic publishing industry, these results suggest LLM reviews should not replace human reviewers but rather augment them in targeted ways. Publishers investing in AI review infrastructure should implement structured prompting techniques while remaining cautious about information retrieval augmentation. The research also implies that relying solely on LLM-generated reviews could systematically overlook experimental concerns, potentially affecting which papers get published. Organizations developing peer review tools should incorporate these findings to ensure AI assistance maintains scientific standards rather than reducing them.

• →LLMs and humans evaluate academic papers using different criteria, with LLMs prioritizing theory over experimental methodology
• →Chain-of-Thought prompting substantially improves LLM review quality, offering a practical enhancement technique
• →Retrieval-augmented generation produces inconsistent results across different LLMs and sometimes degrades review quality
• →LLM-generated reviews should augment rather than replace human peer review to maintain scientific integrity
• →The framework provides datasets and insights for developing more human-aligned AI review systems