CULTURESCORE: Evaluating Cultural Faithfulness in Video Generation Models

Researchers introduce CultureScore, a new evaluation framework for assessing cultural faithfulness in video generation models, revealing that leading AI systems like Veo 3.1 and LTX-2 fail to accurately represent diverse global cultures. Testing across 10 countries shows the best model achieves only 56.8% cultural accuracy, with human evaluators valuing cultural representation over visual quality metrics.

The emergence of CultureScore addresses a fundamental gap in AI model evaluation: while existing metrics like VideoScore measure visual fidelity, they ignore cultural accuracy entirely. This blind spot creates perverse incentives where models that replace culturally specific gestures with Western alternatives receive identical scores as culturally faithful systems. The research demonstrates that current state-of-the-art video generation models perform poorly across all three dimensions tested—Identity, Context, and Behavior—with Behavior being particularly problematic below 52% across all models.

This work reflects broader tensions in AI development where technical advancement outpaces responsible deployment considerations. As video generation becomes more accessible and commercially deployed, the risk of systematic cultural misrepresentation scales dramatically. The framework's finding that human annotators inverted their preferences relative to VideoScore highlights a critical disconnect: users intuitively value cultural authenticity but evaluation systems don't measure it.

For the AI industry, CultureScore creates accountability pressure. Developers cannot claim their models are production-ready without addressing cultural faithfulness, particularly for applications serving global audiences or marginalized communities. The compositional framework is replicable and extensible, likely inspiring similar evaluation suites for other modalities and domains. This establishes new standards that could reshape model development priorities.

The path forward involves integrating cultural evaluation into standard model benchmarking and potentially requiring cultural fairness disclosures similar to existing safety documentation. Organizations building video generation systems now face pressure to benchmark against CultureScore metrics, potentially requiring significant architectural changes or training data improvements.

• →Current video generation models fail cultural representation tests, with best performers achieving only 56.8% accuracy on new CultureScore benchmark
• →Behavioral accuracy (gestures, interactions) is the most challenging dimension for AI, remaining below 52% across all tested models
• →Existing visual quality metrics like VideoScore are inadequate and can mask cultural inaccuracies in generated content
• →Human evaluators prioritize cultural faithfulness over visual quality, indicating misalignment between current metrics and user preferences
• →CultureScore framework spanning 10 countries and 6,180 videos establishes new standards for equitable AI model evaluation