RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension

Researchers introduce RPC-Bench, a large-scale benchmark containing 15,000 human-verified question-answer pairs designed to evaluate how well AI models understand research papers. Testing reveals that even the strongest models like GPT-5 achieve only 68.2% accuracy on comprehension tasks, dropping significantly when conciseness is factored in, exposing critical gaps in academic document understanding.

RPC-Bench addresses a genuine limitation in current AI evaluation frameworks by establishing rigorous metrics for assessing research paper comprehension. Unlike existing benchmarks that offer limited evaluation depth, this benchmark leverages review-rebuttal exchanges from high-quality computer science papers to create authentic, contextually rich assessment scenarios. The taxonomy aligns with actual scientific workflows, testing models on explanatory (why), factual (what), and procedural (how) questions that reflect real academic discourse.

The research emerges as foundation models increasingly serve academic and scientific communities, yet their performance on specialized technical content remains poorly understood. By grounding evaluation in peer-review dynamics, the benchmark captures nuances that generic QA datasets miss—including complex figures, tables, and domain-specific terminology. The LLM-as-a-Judge evaluation framework represents methodological advancement, enabling scalable assessment while maintaining quality control through human verification.

Results carry significant implications for AI development priorities. GPT-5's 68.2% baseline on correctness-completeness, degrading to 37.46% under conciseness constraints, reveals that models struggle with both accurate understanding and precise articulation of complex concepts. This gap matters for organizations building AI-assisted research tools, literature review systems, and academic applications where accuracy directly impacts user trust and utility.

Developers and researchers should monitor whether leading AI labs subsequently optimize their models against RPC-Bench metrics, as adoption could drive architectural improvements in handling technical documentation. The open-source availability of benchmark data creates opportunities for competitive model development, potentially accelerating improvements in scientific document comprehension across the entire AI ecosystem.

• →RPC-Bench contains 15,000 verified QA pairs from computer science peer-review exchanges, providing fine-grained evaluation of research paper comprehension.
• →Even GPT-5 achieves only 68.2% correctness on the benchmark, dropping to 37.46% when conciseness is required, exposing significant limitations in academic understanding.
• →The benchmark uses a scientific workflow-aligned taxonomy to assess why, what, and how questions reflecting authentic scholarly contexts.
• →An LLM-human interaction annotation framework enables scalable evaluation with high agreement to human judgment.
• →Open-source release of code and data creates competitive opportunities for model optimization on specialized technical comprehension tasks.