ReviewGuard: Aligning LLM-Assisted Peer Review with Long-Term Scientific Impact

Researchers introduce ReviewGuard, an LLM-based framework that predicts long-term scientific impact rather than mimicking human peer reviewers. Testing on 20,861 AI/ML papers shows ReviewGuard correlates 5.6x better with future citations than human reviewers and identifies high-impact rejected papers at significantly higher rates, suggesting AI can complement editorial decision-making without replacing human judgment.

ReviewGuard addresses a fundamental inefficiency in academic publishing: peer review systems optimize for contemporary reviewer preferences rather than identifying work with lasting scientific value. The framework uses reinforcement learning to align LLM-generated assessments with citation-based impact metrics, creating a predictive signal distinct from traditional peer evaluation. This approach leverages the growing availability of citation data and computational linguistics to solve a problem that has plagued scientific communities for decades.

The research builds on recent progress in LLM applications across knowledge work. While previous efforts focused on automating reviewer tasks or replicating human judgments, ReviewGuard pivots toward impact prediction. The substantial performance gap—0.776 versus 0.492 Spearman correlation with future citations compared to humans—suggests systematic biases in human review processes that systematically undervalue certain research directions, methodologies, or emerging fields. The 5.6x improvement in flagging high-impact rejected papers has direct institutional implications.

For scientific publishing and research institutions, ReviewGuard offers editors a complementary decision-support tool rather than replacement automation. This positions it as a safety mechanism against type-II errors in peer review, where valuable work faces rejection. The framework's design respecting human editorial authority makes institutional adoption more feasible than fully automated systems. However, the system's performance depends on historical citation data, which may reflect existing biases in what gets cited rather than true scientific merit, a limitation the research doesn't fully address.

• →ReviewGuard achieves 0.776 correlation with long-term citations versus 0.492 for human reviewers on AI/ML papers
• →The system identifies 5.6x more high-impact rejected papers than human reviewers under equivalent decision thresholds
• →Impact-aligned reinforcement learning can augment rather than replace human scientific judgment in peer review
• →LLM-based review systems optimized for future impact outperform those mimicking contemporary reviewer preferences
• →The framework identifies systematic gaps in human peer review, particularly for undervalued research directions