Label Over Logic? How Source Cues Bias Human Fallacy Judgments More Than LLMs

A research study comparing human and LLM reasoning capabilities found that humans are significantly more biased by source labels when evaluating logical fallacies, while LLMs maintain more consistent performance regardless of whether content is attributed to humans or AI. This finding suggests LLMs could enhance human decision-making in AI-mediated environments by providing source-agnostic analysis.

This arXiv research examines a critical vulnerability in human cognition as AI-generated content becomes ubiquitous online. The study demonstrates that humans systematically downgrade their critical evaluation when content appears human-authored, even when the underlying logic contains clear fallacies. Participants assigned higher trust ratings to human-attributed statements compared to AI-attributed ones, despite identical logical quality—a pattern that persists even when humans receive disclosure about AI assistance.

The research addresses a growing concern in digital information ecosystems. As AI tools proliferate, source attribution increasingly influences perception more than actual content quality. This bias has immediate consequences for content moderation platforms, which rely on human judgment to flag misinformation and logical fallacies. If humans systematically trust human-generated content more credibly, they may inadvertently allow fallacious reasoning to spread while over-scrutinizing AI outputs.

LLMs in this study demonstrated remarkable source-agnosticism, maintaining stable performance across all disclosure conditions. This capability positions AI systems as potential correctives for human bias in reasoning evaluation. The implications extend beyond academic research into practical applications: human-AI teams evaluating claims or arguments could leverage AI's source-blind assessment to counter cognitive biases. The study suggests LLMs are less vulnerable to the heuristics that compromise human judgment.

Future research should examine whether this advantage persists across different domains and fallacy types, and whether users can overcome source-label bias through training. The findings also raise questions about AI model transparency—if users distrust AI simply due to its origin, can source-agnostic evaluation actually influence decision-making?

• →Humans assign higher credibility and trust to human-attributed content even when it contains logical fallacies, while LLMs evaluate reasoning quality consistently regardless of source labels.
• →Source-label bias appears primarily a human vulnerability, not shared by current LLM systems tested across GPT, Gemini, and Claude models.
• →Human confidence levels remain high across all source conditions despite fallacy presence, indicating overconfidence independent of reasoning quality.
• →Human-LLM collaboration could mitigate reasoning evaluation biases by pairing human judgment with AI's source-agnostic assessment capabilities.
• →This research highlights a critical gap in human cognition that becomes increasingly problematic as AI-generated content fills online spaces.