Learning to Negotiate: Multi-Agent Deliberation for Collective Value Alignment in LLMs

Researchers propose a multi-agent negotiation framework for aligning large language models in scenarios involving conflicting stakeholder values. The approach uses two LLM instances with opposing personas engaging in structured dialogue to develop conflict resolution capabilities while maintaining collective agency alignment.

• →Multi-agent negotiation framework addresses LLM alignment limitations in multi-stakeholder scenarios with conflicting values.
• →Two self-play LLM instances with opposing personas engage in turn-based dialogue to synthesize mutually beneficial solutions.
• →Training uses synthetic moral-dilemma prompts and RLAIF optimization with GRPO and external LLM reward models.
• →Results show improved conflict-resolution performance while maintaining general language capabilities.
• →Approach provides practical path toward LLMs that better support collective decision-making in value-conflict scenarios.