When Single Answer Is Not Enough: Rethinking Single-Step Retrosynthesis Benchmarks for LLMs

Researchers propose a new benchmarking framework for evaluating large language models in retrosynthesis planning, introducing ChemCensor—a metric prioritizing chemical plausibility over exact-match accuracy—and CREED, a dataset of millions of validated reaction records that improves model performance beyond existing LLM baselines.

The article addresses a critical gap in how the scientific community evaluates LLMs for drug discovery applications. Current benchmarking methods rely on Top-K accuracy metrics that demand exact matches to published procedures, an approach that fails to reflect real-world synthesis planning where chemists routinely identify multiple viable pathways. This limitation has created a false ceiling on LLM performance evaluation, potentially underestimating their practical utility in pharmaceutical research.

The shift from single-answer benchmarks to plausibility-based evaluation represents a maturation in how AI systems are assessed for domain-specific tasks. Retrosynthesis—the process of identifying starting materials and reactions to synthesize target compounds—is inherently open-ended, making rigid accuracy metrics misaligned with laboratory practice. By introducing ChemCensor, the researchers provide a metric that evaluates whether proposed reactions are chemically sound rather than requiring perfect historical precedent, fundamentally changing how LLM contributions to chemistry are measured.

The introduction of CREED, a dataset containing millions of validated reactions, establishes a training resource that could accelerate LLM development for drug discovery workflows. This infrastructure investment signals growing institutional commitment to AI-assisted synthesis planning, with implications for pharmaceutical companies seeking to streamline their discovery pipelines. Models trained on validated chemical data rather than limited published literature may prove substantially more practical for industry applications.

Looking forward, the adoption of plausibility-based metrics could reshape how specialized LLMs are developed across scientific domains. The framework's success may encourage similar evaluations in materials science, protein folding, and other fields where ground truth solutions are not unique. Researchers and organizations should monitor how rapidly this benchmarking approach gains adoption among drug discovery teams.

• →ChemCensor metric evaluates chemical plausibility rather than exact-match accuracy, better reflecting real-world synthesis planning practices
• →CREED dataset provides millions of validated reaction records for training LLMs in retrosynthesis, establishing new training infrastructure
• →Single-answer benchmarks fail to capture the open-ended nature of chemistry, creating artificial performance ceilings for LLM evaluation
• →Models trained on validated chemical data demonstrate improvements over LLM baselines, validating the new benchmarking approach
• →Plausibility-based evaluation frameworks could reshape AI assessment methodologies across scientific domains beyond chemistry