RWGBench: Evaluating Scholarly Positioning in Related Work Generation

Researchers introduce RWGBench, a new evaluation framework for assessing how well AI language models generate related work sections in academic papers. Unlike existing metrics that measure text similarity, RWGBench evaluates citation selection and scholarly positioning—capturing whether models choose appropriate references and frame them correctly, revealing limitations current systems obscure.

RWGBench addresses a critical gap in how researchers evaluate AI-generated academic content. While large language models demonstrate impressive fluency in scientific writing, existing benchmarks apply summarization metrics that miss domain-specific failures. A model might produce grammatically sound related work sections while making academically damaging errors—citing irrelevant papers, misrepresenting prior research, or failing to properly position a study within its field. This distinction matters significantly because related work sections serve as intellectual scaffolding for papers, establishing credibility and context.

The benchmark draws from 40,108 computer science papers and a retrieval corpus of 1.09 million documents, with human-curated evaluation across 100 papers. Its multi-dimensional framework assesses citation selection, contextual appropriateness, organization, and discourse structure—metrics grounded in actual scholarly practices rather than statistical similarity. Human evaluation demonstrates that citation-centric metrics align substantially better with expert judgment than surface-level text comparison, validating the approach.

For the AI research community, RWGBench provides infrastructure for developing more academically-reliable systems. This matters as universities and researchers increasingly adopt AI writing assistants. A tool generating plausible-sounding but incorrectly-cited content could damage scholarly credibility and propagate misrepresentations of prior work. The benchmark's emphasis on citation-level decision-making sets a precedent for task-specific evaluation frameworks that capture nuanced, domain-critical performance dimensions.

Looking ahead, similar citation-centric evaluation approaches may emerge for other academic writing tasks. The work highlights how domain expertise must shape AI evaluation metrics, particularly where generic text similarity metrics fail to capture meaningful quality differences.

• →RWGBench evaluates related work generation through citation selection and scholarly positioning rather than text similarity metrics
• →Current AI systems generate fluent text while making academically critical failures that standard metrics fail to detect
• →Human evaluation shows citation-centric metrics align substantially better with expert judgment than surface-level comparison
• →Framework uses 40,108 papers and 1.09 million documents to establish domain-specific evaluation standards
• →Work establishes precedent for task-specific evaluation frameworks in academic AI applications