MIRA: A Bilingual Benchmark for Medical Information Response Audit

Researchers introduced MIRA, a bilingual benchmark testing whether large language models provide consistent medical information across different user phrasings, health literacy levels, and languages. The study revealed that LLMs systematically omit key medical details when responding to low-health-literacy queries, a pattern termed Differential Information Dilution (DID), with implications for equitable health information access.

MIRA addresses a critical gap in LLM safety evaluation by measuring whether AI systems maintain information quality across demographic and linguistic variables. While existing benchmarks assess factual accuracy and safety, they typically use uniform prompts, missing how models adapt responses to perceived user sophistication. The research found that five mainstream LLMs consistently diluted information for low-literacy signals, reducing concrete next steps and supporting guidance—a troubling pattern when LLMs increasingly serve as first-contact health information sources for diverse populations.

This work emerges amid growing scrutiny of AI fairness and health equity. LLMs are deployed in clinical decision support, patient education, and direct-to-consumer health apps, yet their responses shape health behavior across socioeconomic strata. Differential information dilution suggests these systems may inadvertently widen health literacy gaps by providing less actionable guidance precisely to users who need it most.

The findings carry meaningful implications for healthcare technology developers and regulators. Companies integrating LLMs into health platforms must confront whether simplified responses genuinely improve comprehension or inappropriately restrict information access. The bilingual assessment reveals language effects vary by model, suggesting no universal pattern—requiring model-specific auditing. The knowledge-guided mitigation prompt showing 6-8% improvement offers a practical pathway for reducing DID, though incomplete remediation underscores the challenge.

Looking ahead, healthcare regulators may demand fairness audits similar to MIRA before LLM deployment in clinical contexts. This could establish benchmarking standards, drive competitive pressure for equitable models, and influence AI safety compliance frameworks in regulated industries.

• →LLMs systematically provide less medical detail and fewer action steps when responding to low-health-literacy prompts, creating a fairness problem in health information access.
• →Differential information dilution occurs across five mainstream models but manifests with model-specific language effects, requiring tailored evaluation approaches.
• →Knowledge-guided mitigation prompts reduce information dilution by 6-8% for top performers, suggesting feasible but incomplete technical solutions.
• →Healthcare regulators may adopt medical information fairness benchmarks as prerequisite audits before LLM deployment in patient-facing applications.
• →Bilingual evaluation reveals language is not uniformly disadvantaging, enabling model-specific optimization for multilingual health information equity.