MedCalc-Bench Doesn't Measure What You Think: A Benchmark Audit and the Case for Open-Book Evaluation

Researchers audited the MedCalc-Bench benchmark for evaluating AI models on clinical calculator tasks, finding over 20 errors in the dataset and showing that simple 'open-book' prompting achieves 81-85% accuracy versus previous best of 74%. The study suggests the benchmark measures formula memorization rather than clinical reasoning, challenging how AI medical capabilities are evaluated.

• →MedCalc-Bench dataset contains over 20 critical errors including formula inaccuracies and runtime bugs that were identified and fixed.
• →Simple 'open-book' prompting that provides calculator specifications at inference time dramatically improves accuracy from ~52% to 81-85%.
• →The improved results surpass all published approaches including reinforcement learning systems without requiring any model fine-tuning.
• →GPT-5.2-Thinking achieved 95-97% accuracy, establishing an upper bound with remaining errors due to ground-truth issues.
• →The benchmark appears to measure formula memorization and arithmetic precision rather than genuine clinical reasoning capabilities.