ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments

Researchers introduce ScholarQuest, a large-scale benchmark for evaluating AI agents that search academic papers using language models. The benchmark tests agents across 1,000+ computer science topics with four research intent types, revealing that current agentic methods significantly outperform basic retrieval but still achieve only 31-36% recall, exposing substantial performance gaps in AI-driven literature discovery.

ScholarQuest addresses a critical gap in AI evaluation infrastructure by creating the first systematic benchmark for agentic academic paper search. As language models increasingly power knowledge discovery workflows, the ability to rigorously measure search agent performance becomes essential for both researchers and institutions relying on these systems. The benchmark's construction from real computer science topics and diverse research intents reflects genuine user behaviors, moving beyond synthetic test cases that often fail to capture real-world complexity.

The research landscape has shifted toward agent-based architectures that iterate and refine queries, mimicking how experienced researchers explore literature. Traditional single-shot retrieval baselines no longer represent the frontier, making ScholarQuest's comparative analysis particularly valuable. The provision of ScholarBase, a shared retrieval backend, ensures reproducibility across teams—a critical foundation for collaborative AI development.

The performance metrics reveal sobering realities for current systems. Achieving only 0.314 Recall@100 means agents miss roughly 69% of relevant papers in their top 100 results, a limitation with direct consequences for research quality and completeness. This performance ceiling affects universities, research institutions, and commercial platforms developing AI-assisted discovery tools. The benchmark's multi-dimensional evaluation—including search efficiency, intent-level robustness, and failure case analysis—provides developers concrete directions for improvement rather than opaque aggregate scores.

Looking forward, this benchmark will likely catalyze competition among AI labs to improve retrieval-augmented generation systems and multi-step reasoning for knowledge work. Organizations building research infrastructure should monitor agent improvements against ScholarQuest metrics as a key indicator of AI readiness for production deployment in academic settings.

• →ScholarQuest enables systematic evaluation of AI agents for academic paper search across 1,000+ topics and four research intent categories.
• →Current best-performing agents achieve only 31-36% recall, indicating substantial room for improvement in agentic search systems.
• →The benchmark includes a shared retrieval backend (ScholarBase) ensuring reproducible evaluation across different research teams.
• →Agentic methods consistently outperform single-shot retrieval baselines, validating iterative agent architectures for literature discovery.
• →Multi-dimensional evaluation metrics track search efficiency, intent-level robustness, and failure modes for comprehensive agent assessment.