When Identity Skews Debate: Anonymization for Bias-Reduced Multi-Agent Reasoning

Researchers present a framework to identify and mitigate identity bias in multi-agent debate systems where LLMs exchange reasoning. The study reveals that agents suffer from sycophancy (adopting peer views) and self-bias (ignoring peers), undermining debate reliability, and proposes response anonymization as a solution to force agents to evaluate arguments on merit rather than source identity.

Multi-agent debate represents a promising approach to enhance LLM reasoning quality by aggregating perspectives across multiple agents. However, this research exposes a critical vulnerability: agents make decisions based on who presents an argument rather than the argument's content. The distinction between sycophancy and self-bias is particularly important because it reveals asymmetric failure modes in how LLMs process information within collaborative frameworks.

The formalization of debate dynamics as an identity-weighted Bayesian update process provides theoretical grounding for understanding these biases mechanically. By proposing response anonymization—stripping identity markers from prompts—the researchers create a straightforward intervention that forces equal treatment of all viewpoints. The introduction of the Identity Bias Coefficient offers a quantifiable metric for measuring bias severity across different models and contexts.

For the AI development community, these findings signal that current multi-agent systems may be producing artificially inflated performance gains obscured by identity-driven conformity rather than genuine reasoning improvement. This matters for practitioners building debate-based systems for high-stakes applications like medical diagnosis, legal analysis, or scientific reasoning, where false confidence in erroneous consensus could prove costly.

The empirical finding that sycophancy outpaces self-bias suggests LLMs default to social deference rather than overconfidence, though both undermine system reliability. Moving forward, developers should implement anonymization protocols in debate frameworks and validate that improvements stem from reasoning quality rather than emergent group dynamics. The release of implementation code accelerates community adoption of bias-mitigation techniques.

• →Multi-agent debate systems suffer from identity bias, with agents exhibiting sycophancy (following peers) and self-bias (ignoring peers) based on message source rather than content quality
• →Response anonymization removes identity markers to force equal weighting of arguments regardless of source, reducing bias and improving trustworthiness of debate outcomes
• →The Identity Bias Coefficient provides a quantifiable metric to measure and track identity-driven bias across different LLM models and benchmarks
• →Empirical evidence shows sycophancy is significantly more prevalent than self-bias, indicating LLMs tend toward social conformity over stubborn consistency
• →Current multi-agent reasoning gains may be artificially inflated by identity-driven consensus rather than genuine improvements in logical reasoning quality