CareMedEval dataset: Evaluating Critical Appraisal and Reasoning in the Biomedical Field

Researchers introduce CareMedEval, a new dataset with 534 questions based on 37 scientific articles to evaluate large language models' ability to perform critical appraisal in biomedical contexts. Testing reveals current AI models struggle with this specialized reasoning task, achieving only 0.5 exact match rates even with advanced prompting techniques.

• →CareMedEval dataset contains 534 questions derived from French medical student exams to test AI critical reasoning in biomedicine.
• →State-of-the-art LLMs fail to exceed 0.5 exact match rate on biomedical critical appraisal tasks.
• →Models particularly struggle with questions about study limitations and statistical analysis.
• →Generating intermediate reasoning tokens improves results but doesn't solve fundamental limitations.
• →The benchmark exposes current AI limitations in specialized domain reasoning and critical evaluation.