PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Papers

Researchers introduce PaperScope, a comprehensive benchmark for evaluating multi-modal AI systems on complex scientific research tasks across multiple documents. The benchmark reveals that even advanced systems like OpenAI Deep Research and Tongyi Deep Research struggle with long-context retrieval and cross-document reasoning, exposing significant gaps in current AI capabilities for scientific workflows.

PaperScope addresses a critical evaluation gap in AI research by moving beyond single-document understanding to assess how language models handle realistic scientific workflows. Traditional benchmarks focus on isolated tasks, but modern research requires synthesizing evidence across multiple papers, tables, figures, and text simultaneously—a capability that remains largely unmeasured and underdeveloped. The benchmark's design leverages a knowledge graph of 2,000+ AI papers to create semantically coherent document sets, ensuring that evaluation tasks reflect genuine research complexity rather than artificial scenarios.

The benchmark's significance lies in its systematic exposure of limitations in state-of-the-art systems. Even sophisticated AI platforms specifically designed for deep research—OpenAI Deep Research and Tongyi Deep Research—demonstrate measurable performance gaps when tackling multi-document reasoning at scale. This finding suggests that current approaches to prompt engineering and retrieval-augmented generation may be insufficient for scientific applications requiring deep integration of distributed knowledge. The structural grounding in a knowledge graph provides validity that random document sampling cannot achieve.

For the AI industry, PaperScope establishes a rigorous evaluation standard that will likely influence development priorities. Companies building research-oriented AI tools now face quantifiable benchmarks demonstrating where their systems fall short, creating pressure to invest in improved multi-document reasoning capabilities. The benchmark's open availability should accelerate progress in long-context processing and cross-document synthesis—capabilities essential for applications in science, policy analysis, and competitive intelligence. The scalable pipeline for constructing similar datasets suggests that domain-specific benchmarks across other fields may follow, establishing benchmarking as a competitive differentiator.

• →PaperScope exposes fundamental limitations in advanced AI systems when performing multi-document scientific reasoning tasks
• →The benchmark integrates knowledge graphs and semantic density measures to ensure evaluation reflects realistic research workflows
• →Even purpose-built deep research platforms achieve limited scores, indicating significant technical gaps in long-context retrieval
• →Multi-modal, multi-document benchmarking could become a critical evaluation standard for research-oriented AI development
• →The scalable benchmark pipeline enables systematic evaluation across diverse scientific domains and paper collections