Trust, but Don't Verify: Epistemic Blind Spots in LLM Source Evaluation

A new study reveals that large language models can identify fabricated statistics in isolation but fail to apply this capability when synthesizing multiple sources, instead weighting sources based on analytical presentation style rather than numeric validity. This 'epistemic alignment' failure—where models prioritize how credible something sounds over whether it's actually true—persists across multiple model families and domains, with attempted fixes through prompting producing blanket skepticism rather than selective discernment.

Researchers have uncovered a critical behavioral gap in how language models evaluate evidence: they can detect false statistics when examined alone (76-100% accuracy) but ignore these checks during real-world synthesis tasks. This disconnect matters because LLMs increasingly influence decisions across finance, healthcare, and policy by aggregating information from multiple sources. The models appear to use a 'methodology-register gate'—a learned pattern recognizing the linguistic markers of analytical credibility—that overrides numeric validity signals. Across Claude, Qwen, and OLMo models, impossible confidence intervals receive identical weight as valid ones, suggesting the failure is systematic rather than model-specific. This represents a distinct failure mode from known problems like sycophancy (following user preferences). Instead, models align to epistemic surface markers: they trust what sounds rigorous, not what is logically consistent. The mechanistic analysis using causal tracing and linear probes pinpoints where validity signals exist in model representations but get suppressed during synthesis. Critically, attempted fixes fail—even oracle checklists specifying exact statistical errors trigger blanket rejection rather than targeted corrections. Post-training procedures reinforce stylistic shortcuts instead of building genuine verification capabilities. For enterprise applications relying on LLMs for evidence synthesis, this reveals a fundamental alignment problem orthogonal to capability. Models possess the technical skills but deploy them selectively based on learned heuristics. The implications extend beyond accuracy: if LLMs systematically trust polished misinformation over awkwardly-presented truth, they risk becoming distribution vectors for carefully-crafted falsehoods across professional domains.

• →LLMs can detect fabricated statistics in isolation but systematically fail to apply this capability during multi-source synthesis tasks.
• →Model source-weighting is governed by linguistic presentation style ('methodology-register') rather than actual numeric validity or internal consistency.
• →The failure replicates across three major model families (Claude, Qwen, OLMo) and multiple professional domains, indicating a systematic architectural issue.
• →Standard mitigation approaches including oracle checklists produce blanket skepticism rather than selective discernment, suggesting the problem is a deployment misalignment rather than a capability gap.
• →This 'epistemic alignment' failure creates risks for enterprise applications relying on LLMs for evidence synthesis, particularly in finance and healthcare.