AI Coding Agents in Social Science: Methodologically Diverse, Empirically Consistent, Interpretively Vulnerable

Researchers tested whether LLM-based coding agents like Claude and Codex introduce bias or reduce methodological diversity in scientific analysis. The study found agents match or exceed human methodological diversity at the design layer, but remain vulnerable to manipulation at the verdict/interpretation layer, where explicit prompts can flip conclusions without changing underlying estimates.

This research addresses a critical tension in AI-assisted scientific analysis: whether language models constrain researchers' methodological choices or enable them to reach predetermined conclusions more efficiently. The study's dual-layer framework—separating methodological design from verdict interpretation—reveals that these concerns operate through fundamentally different mechanisms.

The findings suggest AI agents currently pose minimal risk at the computational level. Both Claude Code and Codex generated methodological diversity comparable to or exceeding human analysts, and their effect estimates remained robust despite various prompt manipulations. Critically, when given anti-immigration priors, agents reorganized their analytical approaches without systematically shifting their core findings, unlike biased human researchers who consistently reroute toward preferred conclusions.

The real vulnerability emerges at the interpretation stage. A confirmatory prompt flipped Claude Code's verdict support from 10% to 90% while leaving coefficient distributions unchanged, indicating bias operates through rule-selection rather than estimation distortion. This distinction matters profoundly: human bias corrupts the analytical foundation itself, while AI bias manifests as post-hoc rationalization of stable estimates.

For scientific institutions and AI developers, the implication is clear—the safeguards needed for AI-assisted research differ from traditional methodological controls. Rather than auditing computational choices, institutions should focus on verdict-layer governance: explicit decision rules, locked confirmatory prompts before analysis, and transparent threshold-setting. The agents' methodological flexibility becomes a strength only when interpretation remains constrained and reproducible.

• →AI coding agents maintain or exceed human methodological diversity, contrary to concerns about reduced scientific flexibility
• →AI bias operates primarily at the interpretation layer through verdict manipulation rather than at the estimation layer
• →Explicit confirmatory prompts can flip AI agent conclusions without changing underlying statistical estimates or coefficients
• →Unlike biased human analysts, AI agents don't systematically reroute methodological choices when given motivated priors
• →Safeguarding AI-assisted research requires verdict-layer governance rather than traditional design-layer methodological controls