DeliberationBench: A Normative Benchmark for the Influence of Large Language Models on Users' Views

Researchers developed DeliberationBench, a new benchmark to assess how large language models influence users' opinions on policy matters. A study of 4,088 participants discussing 65 policy proposals with six frontier LLMs found that these models have substantial influence that appears to align with democratically legitimate deliberative processes.

• →DeliberationBench uses deliberative opinion polling as a standard to distinguish beneficial from harmful LLM influence on user beliefs.
• →The study involved 4,088 U.S. participants discussing 65 policy proposals with six frontier LLMs in a randomized experiment.
• →LLMs showed substantial influence on users' opinions that positively correlated with shifts seen in traditional deliberative polling.
• →The research suggests tested LLMs produce epistemically desirable effects rather than harmful manipulation.
• →The framework can serve as an evaluation tool to ensure LLM influence remains consistent with democratic standards.