ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences

Researchers introduce ReplicatorBench, a comprehensive benchmark for evaluating AI agents' ability to replicate scientific research claims in social and behavioral sciences. The study reveals that current LLM agents excel at designing and executing experiments but struggle significantly with data retrieval, highlighting critical gaps in autonomous research validation capabilities.

ReplicatorBench addresses a fundamental gap in AI agent evaluation by moving beyond simple code reproduction to test whether LLM agents can perform genuine scientific replication—a more complex task requiring data discovery, experimental design, and result interpretation. The benchmark's inclusion of both replicable and non-replicable research claims represents a methodological advance, enabling agents to demonstrate not just computational competence but judgment about research validity.

The emergence of AI agents for scientific assessment reflects broader trends in automating knowledge work and quality control across institutions. Scientific reproducibility has long plagued academia, with numerous high-profile studies failing replication attempts. Introducing AI agents capable of independent verification could transform peer review and meta-science practices, though the ReplicatorBench findings suggest current systems remain preliminary.

For the research and academic sectors, these results indicate that AI agents cannot yet replace human replicators in critical roles. The specific weakness in data retrieval—finding necessary datasets for replication—suggests that knowledge graph limitations and web search capabilities remain bottlenecks. This has practical implications for academic institutions considering AI-assisted peer review systems; current LLMs would require substantial human oversight.

The public availability of ReplicatorBench and ReplicatorAgent code democratizes evaluation standards for research validation AI, likely spurring competitive improvements. Future development priorities should focus on enhancing information retrieval capabilities and building agents that better handle data scarcity scenarios common in social sciences. Institutions investing in research integrity tools should monitor LLM agent improvements closely, as capabilities could shift rapidly.

• →LLM agents successfully design and execute computational experiments but fail consistently at retrieving new data needed for replication.
• →ReplicatorBench introduces human-verified ground truth including non-replicable claims, enabling evaluation of agents' judgment beyond technical execution.
• →Current AI agents cannot autonomously replicate scientific claims without human supervision, limiting immediate applications in peer review.
• →Data retrieval limitations represent the primary technical barrier preventing AI agents from performing independent research validation.
• →Open-source availability of benchmark and tools will accelerate development of improved research validation AI systems.