Closed-loop Auto Research for Molecular Property Prediction: Discovering and Certifying Generalizable Improvements

Researchers demonstrate that closed-loop automated machine learning systems can discover generalizable improvements in molecular property prediction by having language-model agents modify features, models, and acquire external evidence. Testing across 36 molecular endpoints reveals that while some improvements validate strongly, they don't consistently transfer to held-out test sets, highlighting critical challenges in ensuring reproducibility of AI-driven research discoveries.

This research addresses a fundamental challenge in automated machine learning: the gap between validation performance and real-world generalization. The team's closed-loop Auto Research system uses language-model agents to autonomously modify machine learning pipelines, representing a shift from passive model fitting to active research workflow optimization. Across three major benchmark suites with 36 molecular endpoints, they achieved held-out test improvements ranging from 0.011 to 0.042, demonstrating that some discoveries do generalize beyond the validation signals that selected them.

The work exposes critical failure modes in automated research pipelines. A model-search configuration that improved validation performance by 0.041 degraded to just 0.003 on held-out tests, while curated external data showed negative transfer (-0.019 on test despite 0.022 on validation). The researchers implemented contamination filters rejecting test-overlapping data sources, a necessary but insufficient condition for ensuring genuine transfer. Notably, their automated agent succeeded where matched AutoML controls failed, achieving 0.042 versus 0.006 on certain interventions.

For the AI and chemistry communities, this research establishes a methodological template for validating autonomous discovery systems. The domain-agnostic lesson—separating discovery from held-out certification—applies broadly to any closed-loop system optimizing proxy objectives. The competitive performance against an 84M-parameter pretrained 3D model suggests efficient alternatives to massive foundation models. However, the pervasive gap between validation and test performance signals that autonomous research agents require substantially more rigorous validation frameworks before deployment in high-stakes applications like drug discovery.

• →Closed-loop AI agents can discover generalizable improvements in molecular property prediction, but validation metrics frequently mispredict held-out performance
• →Curated external data provides significant gains for specific tasks (0.17 improvement on CYP2C9) only when contamination filtering removes overlapping test structures
• →Model-search interventions by language-model agents outperformed matched AutoML controls, suggesting code-level modifications enable discoveries beyond standard hyperparameter optimization
• →Improvements vary dramatically by benchmark suite and molecular endpoint, indicating that transferable axes differ across domains requiring adaptive validation strategies
• →Separating discovery from held-out certification is essential for any closed-loop system optimizing proxy metrics, establishing a domain-agnostic validation framework