CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectives

Researchers introduce CLASH, a dataset of 345 high-stakes dilemmas with 3,795 diverse perspectives, revealing that leading language models including GPT-4 and Claude struggle significantly with ambivalent value-based decisions. The study exposes fundamental limitations in LLM reasoning about conflicting values, with top models achieving only 24-51% accuracy on ambivalent scenarios, indicating a critical gap in AI systems designed for high-consequence decision-making.

The CLASH dataset addresses a consequential blind spot in AI evaluation: how language models navigate genuine moral and ethical dilemmas where no objectively correct answer exists. Current benchmarks typically feature clear solutions, making this research uniquely valuable for understanding how AI systems perform when values conflict irreconcilably. The findings are sobering for organizations deploying LLMs in advisory capacities, particularly in domains like healthcare, law, and public policy where stakeholders hold legitimately competing interests.

The research reveals that advanced reasoning capabilities, which excel in mathematics and game theory, do not transfer to value reasoning contexts. Instead, LLMs exhibit distinct failure patterns including early commitment to initial framings and overcommitment to chosen perspectives. This suggests that scaling model capacity or implementing chain-of-thought reasoning alone won't resolve these deficiencies. The discovery that psychological discomfort—a human signal of ambivalence—remains largely incomprehensible to LLMs highlights how AI systems lack crucial meta-awareness during high-stakes judgment.

The steerability findings carry practical implications: models can be directed toward specific values, yet this susceptibility correlates with their underlying value preferences, suggesting systematic bias rather than true flexibility. Third-party perspective reasoning yields better steerability than first-person framing, except for safety-critical domains where personal stakes improve reasoning quality. These nuanced results suggest that deploying LLMs for multi-stakeholder decision support requires careful architectural choices around perspective framing and explicit value transparency.

Future work must focus on developing LLM architectures that genuinely grapple with incommensurable values rather than defaulting to simplified optimization. The gap between how humans and current AI systems handle moral ambiguity will likely remain a critical limitation for years.

• →GPT-4 and Claude-4-Sonnet achieve only 24-51% accuracy on ambivalent high-stakes dilemmas, revealing fundamental LLM limitations in value reasoning.
• →Advanced reasoning strategies effective for math and games fail to transfer to ethical decision-making, creating distinct new failure modes.
• →LLM steerability toward specific values correlates with underlying model biases, raising concerns about genuine flexibility versus systematic preference.
• →Third-party perspective framing improves LLM reasoning across most values except safety, where first-person reasoning proves more effective.
• →Current LLMs inadequately comprehend psychological discomfort and temporal value shifts, suggesting architectural limitations beyond scale.