LiveCultureBench: a Multi-Agent, Multi-Cultural Benchmark for Large Language Models in Dynamic Social Simulations

Researchers introduce LiveCultureBench, a new benchmark that evaluates large language models as autonomous agents in simulated social environments, testing both task completion and adherence to cultural norms. The benchmark uses a multi-cultural town simulation to assess cross-cultural robustness and the balance between effectiveness and cultural sensitivity in LLM agents.

• →LiveCultureBench is a new multi-cultural benchmark for evaluating LLM agents in dynamic social simulations beyond just task success.
• →The benchmark simulates a diverse town environment where LLMs must balance task completion with adherence to socio-cultural norms.
• →The research examines cross-cultural robustness of LLM agents and their ability to navigate cultural sensitivities.
• →The study evaluates when LLM-as-a-judge systems are reliable versus when human oversight is needed for evaluation.
• →The benchmark addresses a gap in current LLM evaluations that focus primarily on task success rather than cultural appropriateness.