Arguments that Alter Minds: LLM Rationales Sway Human (and LLM) Notions of Plausibility

Researchers found that LLM-generated arguments significantly influence both human and AI plausibility judgments on commonsense reasoning tasks, with supportive rationales increasing confidence and opposing ones decreasing it. This reveals both a novel tool for studying human cognition and a concerning vulnerability: AI systems can persuade people to doubt their own common sense reasoning.

This research exposes a critical gap between perceived and actual human judgment independence. The study collected over 16,000 plausibility assessments across humans and LLMs, demonstrating that LLM-generated rationales consistently shifted judgment outcomes. When presented with arguments supporting an answer, humans rated it more plausible; opposing arguments decreased ratings. LLMs showed identical susceptibility patterns, suggesting shared vulnerabilities in reasoning frameworks.

The findings emerge amid growing concerns about AI's role in information environments. Prior work identified LLM tendency toward plausible-sounding but false outputs; this study demonstrates that both humans and machines struggle to maintain independent assessment when presented with superficially credible counterarguments. The research is particularly significant because commonsense reasoning represents a domain where humans theoretically possess intrinsic expertise and intuitive confidence.

For AI development and deployment, these results underscore the fragility of human-AI collaborative systems. Applications relying on human verification of AI outputs—medical diagnosis assistance, legal document review, financial analysis—may face compromised oversight if users lack defenses against persuasive LLM rationales. The parallel vulnerability in LLMs themselves raises questions about multi-stage AI reasoning pipelines, where LLMs might persuade downstream systems through generated arguments rather than logical validity.

Looking ahead, the research suggests urgent need for training interventions strengthening epistemic resilience. Developers should explore techniques inoculating users against argument-driven judgment shifts, while researchers investigate whether humans can learn to distinguish genuine reasoning from persuasive-but-hollow LLM outputs. Understanding these influence mechanisms becomes critical as LLMs integrate deeper into decision-support systems.

• →LLM-generated arguments measurably shift both human and AI plausibility judgments despite domain expertise
• →Supporting rationales increased human confidence while opposing ones decreased it, showing systematic persuasive influence
• →The vulnerability exists even in commonsense reasoning where humans possess inherent expertise and intuition
• →LLMs demonstrate identical susceptibility patterns, suggesting shared reasoning architecture weaknesses
• →Critical implications for human-AI collaborative systems relying on human verification as oversight mechanism