Evaluating Large Language Models in Scientific Discovery

Researchers introduce a scenario-grounded benchmark for evaluating large language models in scientific discovery, revealing significant performance gaps compared to general science benchmarks. The framework tests LLMs across biology, chemistry, materials, and physics through project-level tasks involving hypothesis generation and experimental design, showing that current models remain distant from achieving general scientific superintelligence despite demonstrating promise in specific applications.

The introduction of this scientific discovery evaluation (SDE) framework addresses a critical gap in how LLMs are assessed for real-world scientific work. Traditional benchmarks measure decontextualized factual knowledge, but scientific discovery requires iterative reasoning, creative hypothesis generation, and sophisticated result interpretation. By grounding evaluation in genuine research projects defined by domain experts, this work provides a more meaningful assessment of LLM capabilities than existing alternatives.

The research reveals sobering findings about current model capabilities. Despite impressive performance on general science benchmarks, state-of-the-art LLMs show consistent performance degradation when evaluated on discovery-relevant tasks. Notably, scaling model size and enhanced reasoning capabilities demonstrate diminishing returns, suggesting that current architectural approaches may have fundamental limitations for scientific discovery work. The variation in which models perform best across different research scenarios indicates that no single LLM has achieved the kind of general scientific competence that would justify claims of superintelligence.

These findings carry important implications for the AI development community and organizations investing in LLM-driven scientific research. Organizations currently deploying LLMs for drug discovery, materials science, or physics research should temper expectations about autonomous discovery capabilities. The research also suggests that guided exploration and human oversight remain essential components of effective scientific workflows. The SDE framework itself represents valuable infrastructure for the community, enabling reproducible evaluation and tracking progress toward more capable scientific reasoning systems. Looking forward, the framework's design provides clear targets for LLM improvement, particularly in systematic weaknesses shared across different models, offering researchers concrete directions for architectural and training innovations.

• →Current LLMs show significant performance gaps on discovery-focused tasks compared to general science benchmarks, indicating existing evaluations overestimate scientific capabilities.
• →Scaling model size and reasoning approaches show diminishing returns for scientific discovery, suggesting architectural limitations in current LLM designs.
• →No single state-of-the-art model consistently outperforms others across diverse research scenarios, indicating all current LLMs are far from general scientific superintelligence.
• →LLMs demonstrate unexpected promise in scientific discovery despite low individual scenario scores, highlighting the importance of guided exploration in research workflows.
• →The SDE framework provides reproducible benchmarks for evaluating discovery-relevant LLM capabilities and identifies concrete improvement targets for future model development.