Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook

Researchers introduce DOVE, a distributional evaluation framework that measures how well large language models align with cultural values through open-ended text generation rather than multiple-choice tests. The framework uses rate-distortion optimization to create a value codebook and unbalanced optimal transport to assess alignment, demonstrating 31.56% correlation with downstream tasks across 12 LLMs while requiring only 500 samples per culture.

DOVE addresses a fundamental limitation in how the AI industry currently evaluates cultural value alignment in large language models. Existing benchmarks rely on discriminative multiple-choice formats that test knowledge of cultural values rather than genuine alignment, fail to account for subcultural diversity, and don't reflect how LLMs actually generate text in real-world deployment scenarios. This gap between evaluation methodology and practical application has created blind spots in understanding whether globally deployed models genuinely respect diverse cultural orientations.

The framework's innovation lies in its mathematical rigor and practical design. By constructing a compact value-codebook from 10,000 documents using rate-distortion optimization, DOVE transforms unstructured text into a structured value space while filtering semantic noise. The use of unbalanced optimal transport to measure alignment captures both overall distributional differences and important within-culture heterogeneity—a critical distinction since cultures contain diverse value systems rather than monolithic orientations.

For AI developers and safety teams, DOVE's demonstrated reliability with just 500 samples per culture offers significant practical advantages, reducing evaluation costs while improving validity. The 31.56% correlation with downstream tasks suggests the framework captures something meaningful about real-world LLM behavior that previous benchmarks missed. This matters as regulators increasingly scrutinize AI safety and cultural appropriateness across markets. Companies deploying global LLMs can use DOVE to identify value misalignment risks before deployment, particularly for underrepresented cultures often overlooked in benchmark construction.

The framework's success in testing 12 different LLMs indicates its generalizability. Going forward, the key challenge lies in integration—whether major AI labs adopt DOVE or similar distributional approaches rather than continuing with cheaper but less valid multiple-choice evaluations.

• →DOVE replaces multiple-choice cultural value testing with distributional analysis of open-ended LLM-generated text, improving predictive validity by measuring actual alignment rather than value knowledge.
• →Rate-distortion optimization creates a compact value-codebook that filters semantic noise while preserving cultural nuance, enabling efficient evaluation with as few as 500 samples per culture.
• →Unbalanced optimal transport measurement captures subcultural heterogeneity within cultures, recognizing that value orientations vary within groups rather than treating cultures as monolithic.
• →The framework achieved 31.56% correlation with downstream tasks across 12 LLMs, outperforming existing benchmarks in predicting real-world cultural value alignment outcomes.
• →Practical evaluation efficiency and improved validity create incentives for adoption in AI safety pipelines, particularly for companies deploying models across diverse global markets.