Polarization by Default: Auditing Recommendation Bias in LLM-Based Content Curation

Researchers audited three major LLM providers (OpenAI, Claude, Google) to assess content curation biases across Twitter/X, Bluesky, and Reddit. The study found that LLMs systematically amplify polarization, exhibit negative sentiment bias, and show political leaning bias favoring left-leaning authors, with varying degrees of mitigation through prompt design.

This research reveals a critical vulnerability in AI-powered content systems that increasingly mediate information access for billions of users. The study's scale—540,000 simulated ranking decisions across 54 experimental conditions—provides robust evidence that content curation bias is not merely incidental but structural to how LLMs operate. The finding that polarization amplifies uniformly across all configurations suggests this isn't a fixable parameter but rather an emergent property of how these models rank human-generated content.

The political leaning bias documented on Twitter/X is particularly significant. Despite right-leaning authors comprising a plurality of the dataset, LLMs consistently over-represented left-leaning voices, indicating biases baked into training data or model architecture that persist even when prompted toward neutrality. This pattern reflects broader concerns about whose values and perspectives become embedded in algorithmic systems that shape public discourse.

The provider comparison reveals different risk profiles: Google's Gemini shows stronger negative sentiment preferences while Claude demonstrates higher adaptivity in handling toxicity. These trade-offs suggest companies face genuine architectural choices rather than unified best practices. For developers and platforms deploying LLMs for content ranking, the research indicates that prompt engineering offers limited mitigation—fundamental biases resist optimization.

This work matters beyond academic interest because content curation directly influences what information reaches users at scale. As platforms increasingly automate editorial functions to LLMs, understanding these systematic biases becomes essential for maintaining healthy information ecosystems. Future research should examine whether these biases affect user behavior and political polarization at population levels.

• →LLMs systematically amplify polarization in content ranking across all tested configurations regardless of prompt strategy.
• →Left-leaning authors are consistently over-represented on Twitter/X despite forming a minority in the dataset, suggesting deep training data biases.
• →Toxicity handling shows inverse behavior between engagement-focused and information-focused prompts, creating an unavoidable trade-off.
• →Different LLM providers exhibit distinct bias profiles, with GPT-4o Mini showing consistency but Gemini showing strongest negative sentiment preference.
• →Prompt design offers limited mitigation for structural biases, suggesting the need for architectural rather than instructional solutions.