From Conversation to Query Execution: Benchmarking User and Tool Interactions for EHR Database Agents

Researchers introduced EHR-ChatQA, a new benchmark for testing AI agents that interact with Electronic Health Record databases through natural language queries. The benchmark reveals significant reliability gaps in current state-of-the-art LLMs, with success rates dropping substantially when consistency across multiple trials is required.

• →EHR-ChatQA benchmark evaluates AI agents on real-world clinical database access workflows including query clarification and SQL generation.
• →State-of-the-art LLMs achieve over 90% Pass@5 success on incremental queries but only 60-70% on adaptive query refinement tasks.
• →Consistency across trials (Pass^5) shows gaps of up to 60%, highlighting reliability issues for safety-critical healthcare applications.
• →The benchmark addresses key challenges of query ambiguity and terminology mismatches between users and database entries.
• →Code and data are publicly available to guide future development of more robust healthcare AI agents.