CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations

Researchers demonstrate that cross-lingual contrastive preference tuning (CroCo) enables large language models to improve performance across 14 languages without language-specific annotations by leveraging English-trained reward models. The method shows consistent gains in both structured and open-ended generation tasks across multiple languages while avoiding catastrophic forgetting.

CroCo addresses a critical challenge in multilingual AI development: scaling preference optimization beyond English without duplicating expensive annotation efforts. The research reveals that reward models trained on English preferences can effectively rank model outputs across typologically diverse languages, enabling efficient transfer learning at scale. This finding has significant implications for developing equitable AI systems that perform well across low-resource languages where preference annotation data remains scarce.

The methodology builds on established principles of contrastive learning in single-language settings, extending them to multilingual contexts through on-policy data generation. Crucially, the authors identify that success depends on generating responses within the target language rather than relying on off-policy data or online optimization—a distinction that informs how preference tuning should be implemented in production systems. This constraint suggests preferences are deeply tied to language-specific generation patterns.

For the AI development community, CroCo offers a pragmatic path forward for multilingual model improvement without proportional increases in annotation costs. The consistent improvements across EuroLLM-9B and Aya-3B models across diverse language families demonstrate reproducibility. However, the mixed results on structured tasks (matching baseline in only 4-6 of 7 languages) indicate the method works better for creative generation than constrained outputs, suggesting refined approaches may be needed for task-specific optimization.

The research opens questions about preference universality across languages and whether multilingual preference tuning could serve as a foundation for emerging language models targeting underserved populations.

• →English-trained reward models successfully rank outputs across 14 diverse languages without language-specific annotations
• →On-policy self-generated data is essential; off-policy responses significantly reduce effectiveness
• →Method prevents catastrophic forgetting while improving performance on 11 of 14 languages for open-ended tasks
• →Structured tasks show more modest gains, suggesting preference tuning effectiveness varies by task type
• →Cross-lingual transfer enables cost-efficient preference optimization for low-resource language models