PaperRepro: Automated Computational Reproducibility Assessment for Social Science Papers

Researchers introduced PaperRepro, a two-stage AI agent system that automates the assessment of computational reproducibility in social science research papers. The system achieved a 21.9% improvement over existing baselines on the REPRO-Bench benchmark by separating code execution from evaluation phases.

• →PaperRepro uses a novel two-stage approach with specialized AI agents for execution and evaluation of research reproducibility.
• →The system addresses key limitations of existing approaches including limited context capacity and inadequate task-specific tooling.
• →Testing on REPRO-Bench showed 21.9% relative improvement in score-agreement accuracy over strongest prior baseline.
• →Researchers created REPRO-Bench-S, a new stratified benchmark for more diagnostic evaluation of automated reproducibility systems.
• →The approach maximizes large language model coding capabilities to enable more complete result capture for evaluation.