ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment

ForeSci introduces a new benchmark for evaluating whether large language model agents can make forward-looking research decisions using only historical evidence, testing 500 tasks across AI domains. The research reveals that while explicit evidence organization improves traceability, a fundamental evidence-decision decoupling problem persists where agents cite relevant sources but reach incorrect conclusions.

ForeSci addresses a critical gap in AI research evaluation: how well LLM agents can make predictive judgments about research directions when future evidence remains unavailable. This matters because AI research strategy—identifying bottlenecks, pursuing promising directions, and positioning projects—inherently requires forward-looking decisions before outcomes materialize. The benchmark's design is rigorous, using temporally controlled cutoffs and pre-cutoff taxonomy branches to prevent agents from pattern-matching future events through training data leakage.

The research emerges from a broader trend of making AI systems more interpretable and reliable as decision-makers. As LLMs become embedded in research workflows and strategic planning, understanding their limitations in forecasting becomes essential. Traditional benchmarks test factual recall or narrow task completion, but ForeSci probes whether agents can synthesize historical evidence into sound prospective judgments—a distinctly harder problem.

For the AI research community, these findings have significant implications. The evidence-decision decoupling problem suggests that citation of relevant sources doesn't guarantee reasoning quality. An agent might correctly identify papers discussing scaling laws but still misforecast which scaling approaches will prove dominant. This has direct consequences for using AI agents in research planning, grant prioritization, and resource allocation across academic and industry labs.

The immediate challenge ahead involves developing agent architectures that not only retrieve relevant evidence but explicitly map how that evidence constrains or supports specific forward-looking claims. Future work should focus on whether agents can quantify confidence levels in predictions and identify evidence gaps that would most reduce uncertainty about research outcomes.

• →ForeSci reveals that LLM agents struggle with evidence-decision decoupling: citing relevant research while forecasting incorrect outcomes
• →Explicit evidence organization through hybrid RAG improves traceability but doesn't uniformly boost forecasting accuracy across decision types
• →The benchmark's temporal cutoff design prevents data-leakage-based future prediction, enabling genuine forward-looking capability assessment
• →Research strategy decisions—bottleneck identification, direction selection, project positioning—require new evaluation frameworks beyond standard benchmarks
• →Agent performance varies significantly by decision family, suggesting domain-specific architectures may be necessary for reliable research forecasting