NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment

Researchers introduced NovBench, the first large-scale benchmark for evaluating how well large language models can assess research novelty in academic papers. The benchmark comprises 1,684 paper-review pairs from a leading NLP conference and reveals that current LLMs struggle with scientific novelty comprehension despite promise in peer review support.

NovBench addresses a critical gap in AI evaluation infrastructure. As academic submission volumes surge, peer review systems face mounting pressure, making computational support increasingly valuable. However, the systematic evaluation of LLMs' novelty assessment capabilities has remained largely absent from research literature. This work fills that void by establishing standardized metrics and datasets for benchmarking.

The benchmark's design reflects nuanced understanding of how academic novelty works in practice. Rather than relying solely on expert reviews, the researchers extracted novelty descriptions directly from paper introductions, recognizing that authors explicitly articulate their contributions in this section. This dual-source approach provides both direct claims and expert evaluation perspectives, creating a more robust evaluation foundation.

The experimental findings carry important implications for the AI-assisted peer review ecosystem. Current models, whether general-purpose or specialized, demonstrate insufficient understanding of scientific novelty and struggle with instruction-following when fine-tuned on review data. This suggests that simply scaling model capacity or domain-specific training is insufficient; targeted architectural or training approaches are needed.

For the academic publishing industry, these results indicate that LLM-assisted peer review remains in early stages. Organizations implementing AI review tools should recognize current limitations and maintain human oversight. The four-dimensional evaluation framework (Relevance, Correctness, Coverage, Clarity) provides publishers and researchers with concrete metrics for assessing future model improvements, potentially driving development of better fine-tuning methodologies.

• →NovBench is the first benchmark specifically designed to evaluate LLM capabilities in assessing academic paper novelty
• →Current LLMs show limited understanding of scientific novelty despite fine-tuning on peer review data
• →Fine-tuned models often fail at instruction-following, indicating scaling alone cannot solve the problem
• →The benchmark uses 1,684 paper-review pairs with a four-dimensional evaluation framework for quality assessment
• →Results suggest targeted improvements to fine-tuning strategies are necessary before LLMs can reliably support human peer review