MedMeta: A Benchmark for LLMs in Synthesizing Meta-Analysis Conclusion from Medical Studies

Researchers introduce MedMeta, a benchmark evaluating how well large language models can synthesize conclusions from medical meta-analyses using only study abstracts. The study reveals that retrieval-augmented generation (RAG) significantly outperforms parametric-only approaches, but all current models struggle with evidence synthesis and fail to properly reject contradictory findings, achieving only marginally above-average performance even under ideal conditions.

MedMeta addresses a critical gap in AI evaluation by moving beyond factual recall benchmarks to test higher-order reasoning capabilities in medical contexts. This research matters because medical applications demand rigorous evidence synthesis—a task where current LLMs demonstrate surprising weakness despite their general capability advances. The benchmark comprises 81 meta-analyses from PubMed spanning 2018-2025, with evaluation across two distinct workflows that isolate the contribution of external information versus internal knowledge.

The findings reveal fundamental architectural limitations in current LLM systems. While retrieval-augmented generation consistently outperforms parametric-only approaches, the margin of improvement plateaus quickly, and crucially, all models fail to identify and reject negated or contradictory evidence—a vulnerability that poses real clinical risks. Even under optimal RAG conditions with ground-truth abstracts, models achieve only ~2.7/5.0 performance, suggesting this isn't a scaling problem but a reasoning problem.

The research challenges the prevailing narrative that domain-specific fine-tuning drives medical AI progress. Instead, the evidence suggests that robustness in grounding systems and evidence handling matters far more than specialized training. For the medical AI industry, this redirects investment priorities toward improving RAG infrastructure rather than pursuing larger or more specialized models.

Looking ahead, the critical vulnerability in evidence rejection presents both a research opportunity and a cautionary tale for clinical deployment. Organizations developing medical applications should view this benchmark as a reality check—current models require substantial architectural improvements in reasoning and negation handling before clinical deployment can be responsibly expanded. The research validates LLM-as-judge evaluation through rigorous correlation analysis, making MedMeta a reusable standard for future development.

• →Retrieval-augmented generation substantially outperforms parametric-only approaches, but all models achieve only marginally above-average performance even with ground-truth abstracts.
• →Current LLMs fail to identify and reject negated or contradictory evidence, representing a critical vulnerability in medical applications.
• →Domain-specific fine-tuning provides minimal benefits when external information is available, suggesting architecture and grounding matter more than specialization.
• →MedMeta comprises 81 real meta-analyses validated against human expert ratings with 0.81 Pearson correlation, establishing it as a reliable evaluation framework.
• →Developing robust retrieval-augmented generation systems is a more promising direction for clinical AI than pursuing larger or more specialized models.