SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations

Researchers introduce SoCRATES, a new benchmark for evaluating how well large language models can mediate conflicts across diverse scenarios and cultural contexts. Testing eight frontier LLMs reveals that even top-performing mediators resolve only about one-third of disagreements, with significant performance variations based on cultural identity, emotional reactivity, and party composition.

SoCRATES addresses a critical gap in AI evaluation methodology by moving beyond static, single-domain testing toward realistic mediation scenarios. Traditional LLM evaluation frameworks rely on expert-authored test cases within narrow domains and treat every response equally, generating noise when models address off-topic elements. This new benchmark constructs conflict scenarios from real-world disputes across eight distinct domains using an automated pipeline, then evaluates responses only on turns that meaningfully advance specific discussion topics, achieving 0.82 alignment with human expert judgments—more than double previous baseline performance.

The benchmark's innovation lies in probing five socio-cognitive adaptation axes: strategic posture, party composition, conversation history length, emotional reactivity, and cultural identity. These dimensions reflect how human mediation varies dramatically across contexts. Testing eight frontier LLMs reveals a sobering finding: even the strongest models close only approximately one-third of the consensus gap that exists in unmediated disputes, indicating substantial room for improvement in LLM mediation capabilities.

This research has implications for developers building AI-assisted conflict resolution tools, whether for workplace disputes, customer service, or diplomatic contexts. The sharp performance variance across socio-cognitive axes suggests that scaling model size alone won't solve mediation challenges—LLMs require social adaptation capabilities that current architectures lack. Organizations implementing AI mediators should expect uneven performance depending on cultural context, party composition, and emotional dynamics. The benchmark itself becomes a valuable tool for researchers optimizing mediation behavior, establishing clearer evaluation standards than previously existed.

• →SoCRATES benchmark uses real-world conflict scenarios across eight domains to evaluate LLM mediation more realistically than prior single-domain approaches.
• →Topic-localized evaluation methodology achieves 0.82 alignment with human experts, more than doubling baseline accuracy by ignoring off-topic noise.
• →Eight frontier LLMs tested resolve only about one-third of unmediated consensus gaps, revealing significant limitations in current mediation capabilities.
• →Performance varies sharply across socio-cognitive axes including cultural identity and emotional reactivity, indicating LLMs need social adaptation rather than just scale.
• →Results suggest AI mediation tools require domain-specific fine-tuning and cultural adaptation strategies to be effective in diverse real-world contexts.