DeliChess: A Multi-party Dialogue Dataset for Deliberation in Chess Puzzle Solving

Researchers introduce DeliChess, a dataset of 107 multi-party dialogue transcripts where groups collaboratively solve chess puzzles through deliberation. The study finds that group discussion significantly improves accuracy, though the role of probing questions in driving performance gains remains inconsistent.

DeliChess represents a meaningful contribution to understanding how humans reason collectively in structured domains. The dataset captures a natural experimental setting—individual puzzle attempts followed by group discussion—that mirrors real-world decision-making processes in professional and academic environments. By grounding analysis in chess, researchers obtain objective performance metrics through engine evaluation, eliminating subjectivity in assessing solution quality.

The work addresses a notable gap in existing research. Most dialogue datasets focus on open-ended conversation or simple information exchange rather than collaborative problem-solving under constraints. Multi-party deliberation datasets are particularly scarce, yet group reasoning appears increasingly relevant as organizations adopt distributed decision-making and cross-functional teams. Chess provides an ideal testing ground because puzzle solutions are verifiable and difficulty is quantifiable.

The finding that deliberation improves group accuracy has practical implications for organizational settings, suggesting structured discussion protocols enhance collective judgment. However, the discovery that probing utterances increase performance variability without consistent gains reveals a counterintuitive dynamic: prompting deeper reflection or justification doesn't automatically improve outcomes. This suggests that dialogue quality matters more than dialogue quantity, and that poorly-directed questioning may introduce noise rather than insight.

Future work should explore which dialogue patterns most reliably predict performance improvements and whether findings generalize beyond chess. The dataset enables research into dialogue modeling, team dynamics, and consensus-building mechanisms. Applications could range from improving AI systems that mediate group decisions to designing better protocols for expert panels and collaborative platforms.

• →DeliChess dataset contains 107 transcripts of group chess puzzle solving with measurable performance metrics before and after deliberation.
• →Deliberation significantly improves group accuracy compared to individual performance, validating the value of collaborative problem-solving.
• →Probing utterances increase variability in group performance rather than reliably boosting accuracy, suggesting dialogue quality is nuanced.
• →Chess provides an objective domain for studying multi-party reasoning and dialogue dynamics with engine-verified solution quality.
• →Dataset offers research infrastructure for modeling group reasoning, consensus resolution, and collaborative decision-making in structured environments.