Seeing Like an AI: How LLMs Apply (and Misapply) Wikipedia Neutrality Norms

Researchers evaluated how large language models detect and correct biased Wikipedia edits according to the Neutral Point of View policy. LLMs achieved only 64% accuracy at bias detection but performed better at correction (79% word-removal accuracy), though they made extraneous changes beyond what human editors would make, revealing tensions between AI effectiveness and community standards.

This research exposes a fundamental challenge in deploying LLMs within specialized communities: models trained on broad internet corpora struggle to internalize nuanced editorial norms even when explicit rules are provided. The study's findings reveal competing failure modes—some models under-detect bias while others over-correct it, suggesting LLMs lack genuine understanding of neutrality principles and instead apply statistical patterns learned from training data.

The performance gap between detection and generation is particularly revealing. While LLMs excelled at removing biased language (79% accuracy), they simultaneously introduced unrelated grammatical changes and stylistic edits that Wikipedia editors avoided. This behavior reflects a fundamental misalignment: LLMs optimize for coherence and fluency rather than minimal, targeted intervention. Crowdworkers preferred AI-generated rewrites, rating them as more neutral and fluent, yet this preference diverges from expert editor judgment—a critical distinction for platforms valuing community authority.

For platforms and organizations considering AI-assisted moderation, this research demonstrates that rule-articulation alone cannot substitute for institutional knowledge. LLMs may inadvertently reshape community norms through their rewrites, increasing moderation burden rather than reducing it. The findings suggest LLMs function best as assistants flagging potential issues rather than autonomous decision-makers applying community policies. As AI systems become more prevalent in content moderation, maintaining human editorial control and preserving community agency emerge as essential safeguards against algorithmic drift.

• →LLMs achieved only 64% accuracy at detecting biased Wikipedia edits despite being provided explicit NPOV policy guidelines.
• →Models performed better at correction tasks but made extraneous changes beyond Wikipedia editors' targeted neutralizations, creating high-recall but low-precision edits.
• →Crowdworkers rated AI rewrites as more neutral and fluent than human editor rewrites, revealing a gap between public preference and expert community standards.
• →Different LLMs exhibited contrasting biases in neutrality detection, suggesting models lack genuine understanding rather than applying learned patterns.
• →AI-assisted moderation may reduce editor agency and increase verification workload rather than streamlining editorial workflows.