Contemporary AI lacks the imagination to diverge or negate in science

A major peer-reviewed study of 6,749 scientists evaluated AI-generated research ideas and found that large language models lack imagination in scientific discovery, struggle to propose null hypotheses, and show weak agreement with human expert judgment. The research reveals significant limitations in AI's ability to accelerate science despite widespread industry optimism.

This comprehensive evaluation represents a critical reality check for the AI-in-science hype cycle. Researchers invited over 121,000 preprint authors to assess follow-up research ideas generated by LLMs, collecting nearly 26,000 expert ratings on novelty, feasibility, and adoption likelihood. The findings expose three fundamental gaps in current AI capabilities that challenge prevailing narratives about accelerated scientific discovery.

The collapse of non-reasoning models into a "hivemind" of similar ideas reflects a core limitation: these systems excel at pattern matching within training data but struggle with genuine intellectual divergence. While reasoning models explore wider hypothesis spaces, none spontaneously generated null hypotheses—a distinctly human scientific practice that tests assumptions rather than confirms them. This gap matters because disconfirming evidence often drives scientific progress more than novel affirmations.

The study also reveals concerning misalignment between what AI produces and what scientists need. Automated evaluators including LLM-as-judge models showed weak correlation with expert judgment, suggesting the community's current validation infrastructure is inadequate. The social sciences proved most challenging for AI, indicating that pluralistic fields requiring contextual interpretation and theoretical flexibility expose AI's brittleness most clearly. Senior social scientists dismissed AI suggestions most harshly, suggesting experience enables better calibration of AI limitations.

The authors' custom Qwen3 reward model trained on human ratings improved performance by 27% over existing approaches, yet still fell short of inter-rater consistency among peer reviewers. This finding underscores that AI remains fundamentally a tool requiring human grounding rather than an autonomous discovery engine. The research suggests investors and institutions should temper expectations around near-term autonomous scientific breakthroughs.

• →LLMs consistently collapse into similar ideas lacking intellectual diversity, undermining claims of accelerated scientific discovery
• →No current AI model spontaneously proposes null hypotheses, a critical gap in scientific methodology
• →Automated AI evaluators show weak agreement with expert judgment across domains
• →Social sciences prove most challenging for AI, suggesting context-dependent fields expose fundamental limitations
• →Custom reward models trained on human ratings outperform standard approaches but remain inferior to peer reviewer consistency