SciPredict: Can LLMs Predict the Outcomes of Scientific Experiments in Natural Sciences?

Researchers introduce SciPredict, a benchmark testing whether large language models can predict scientific experiment outcomes across physics, biology, and chemistry. The study reveals that while some frontier models marginally exceed human experts (~20% accuracy), they fundamentally fail to assess prediction reliability, suggesting superhuman performance in experimental science requires not just better predictions but better calibration awareness.

The SciPredict benchmark addresses a critical gap in AI evaluation: most assessments measure scientific knowledge rather than predictive capability for experimental outcomes. This distinction matters profoundly because predicting what happens in novel experiments represents a genuine frontier where AI could theoretically surpass human researchers. The study's 405 empirical tasks spanning 33 scientific sub-fields provide rigorous evidence that current LLMs remain substantially limited for this application, with accuracies between 14-26% compared to human expert performance around 20%.

The research emerges within broader efforts to quantify LLM capabilities beyond language tasks. Previous benchmarks like MMLU and GSM8K focus on knowledge retrieval and mathematical reasoning, but SciPredict specifically targets prediction under empirical uncertainty—a domain where ground truth comes from real experimental data. This represents a natural progression in AI evaluation methodology as the field matures beyond benchmark saturation.

The findings carry significant implications for AI-assisted scientific research. While some frontier models show marginal advantages, the critical failure is calibration: models cannot reliably distinguish between confident-but-wrong predictions and genuinely reliable ones. Human experts demonstrate strong calibration, achieving 80% accuracy on outcomes they deem predictable without experimentation versus only 5% on unpredictable cases. Models show no such discrimination, maintaining ~20% accuracy regardless of confidence levels. This gap suggests that deploying current LLMs as scientific research assistants could mislead rather than accelerate discovery if practitioners trust model confidence scores.

The benchmark establishes clear metrics for measuring progress toward AI systems genuinely useful in experimental science. Future research must focus on building uncertainty quantification into predictive models, not merely improving raw accuracy scores.

• →Current LLMs achieve 14-26% accuracy predicting scientific experiment outcomes, barely exceeding human expert performance at ~20%.
• →Models fail to calibrate predictions, unable to distinguish reliable from unreliable forecasts regardless of confidence levels.
• →Human experts demonstrate strong calibration, with accuracy ranging from 5% to 80% based on predictability assessment.
• →SciPredict benchmark comprises 405 empirical tasks across 33 specialized fields in physics, biology, and chemistry.
• →Superhuman AI performance in experimental science requires better uncertainty quantification, not just improved prediction accuracy.