Beyond One-shot: AI Agents for Learning in Field Experiments

Researchers demonstrated that tool-augmented AI agents can automatically learn from experimental data to design superior interventions, outperforming human-AI collaboration in a large-scale healthcare field study. The AI-generated messaging achieved 69.8% click-through rates, but results suggest domain-specific experimental data—not general reasoning ability—drives performance.

This research represents a meaningful shift in how organizations can extract value from experimental data. Rather than treating each A/B test as an isolated event, the study demonstrates that AI agents equipped with analytical tools and structured reasoning can identify patterns from prior experiments and autonomously generate improved interventions. The healthcare messaging context involved nearly 700,000 patient visits across two stages, providing substantial evidence that this approach produces measurable gains over traditional human-expert collaboration.

The distinction between general-purpose reasoning and domain-specific learning is critical. While frontier large language models performed poorly without access to actual experimental data, the same models equipped with field results and analytical frameworks generated messages significantly outperforming baseline interventions. This finding challenges assumptions that scaling model capability alone solves complex design problems. Instead, it validates a practical framework where AI serves as a reasoning partner for extracting implicit knowledge from experimental datasets.

For organizations conducting routine testing—whether in healthcare, e-commerce, or product development—this approach offers a scalable alternative to expensive expert consultations. The agentic methodology could accelerate the pace of cumulative learning across multiple experiment cycles. However, the results also expose limitations in existing behavioral theory frameworks when applied to specific contexts, suggesting that field-experiment-driven refinement becomes necessary.

The research trajectory points toward AI systems that function as experimental collaborators rather than one-off solution generators. As organizations accumulate experimental datasets, the value of tool-augmented agents capable of principled reasoning over domain data may exceed that of general-purpose models, reshaping how teams approach evidence-based decision making.

• →Tool-augmented AI agents outperformed human-expert collaboration by autonomously learning from prior experimental data to design superior interventions.
• →Domain-specific experimental data proved more valuable than general reasoning ability, with frontier LLMs failing to predict intervention success without field results.
• →The best AI-generated message achieved 69.8% CTR, a 6.5 percentage point improvement over baseline performance in healthcare contexts.
• →General-purpose behavioral theories do not uniformly extend to specific healthcare contexts, revealing a need for field-experiment-driven theory validation.
• →This approach transforms experimentation from isolated one-shot tests into scalable systems for cumulative design learning across multiple intervention cycles.