EpiQAL: Benchmarking Large Language Models in Epidemiological Question Answering and Reasoning

Researchers introduced EpiQAL, the first benchmark for evaluating large language models on epidemiological reasoning tasks. Testing 15 models reveals significant performance gaps in multi-step inference and evidence synthesis, indicating current LLMs struggle with population-level disease analysis despite their general capabilities.

EpiQAL addresses a critical gap in AI evaluation frameworks by focusing on epidemiological reasoning rather than general clinical knowledge. While existing medical benchmarks test patient-level diagnosis and treatment, epidemiology requires synthesizing disparate studies to understand disease burden, transmission patterns, and intervention effectiveness at population scales. This distinction matters because epidemiological inference demands rigorous evidence integration—a skill increasingly important for public health AI applications.

The benchmark's three-tier architecture progressively tests factual recall, multi-step reasoning, and conclusion reconstruction under incomplete data. This design reflects real-world epidemiological challenges where researchers must work with partial information and competing evidence. The quality-controlled construction pipeline combining taxonomy guidance and multi-model verification ensures the benchmark measures genuine reasoning capability rather than pattern matching or memorization.

The experimental findings reveal uncomfortable truths about current LLM capabilities. Tested models showed limited epidemiological reasoning performance, with multi-step inference emerging as the primary weakness. Notably, model rankings shifted dramatically across subsets, suggesting that scale alone—a primary differentiator in the AI industry—does not guarantee superior reasoning on specialized domains. Chain-of-Thought prompting showed mixed effectiveness, helping with multi-step problems but not uniformly improving performance.

For AI developers and public health institutions, EpiQAL signals that deploying current LLMs for epidemiological analysis requires substantial caution. The benchmark provides diagnostic signals for improving evidence-grounding and inferential reasoning in specialized domains. Future work likely focuses on training approaches that enhance multi-step reasoning in constrained information environments rather than simply scaling model parameters.

• →Current LLMs demonstrate limited capability in epidemiological reasoning despite strong performance on general medical tasks, indicating domain-specific reasoning gaps.
• →Multi-step inference and conclusion reconstruction under incomplete information represent the most challenging aspects of epidemiological question answering for existing models.
• →Model scale does not reliably predict epidemiological reasoning performance, challenging assumptions about capability correlation with parameter count.
• →Chain-of-Thought prompting provides inconsistent benefits across epidemiological reasoning tasks, suggesting specialized prompting strategies are needed.
• →EpiQAL establishes the first standardized benchmark for evaluating and improving LLM performance on population-level health analysis tasks.