An In-Vitro Study on Cross-Lingual Generalization in Language Models

Researchers introduce a controlled experimental framework using procedurally generated languages to study cross-lingual transfer in language models, isolating variables like lexical distance and tokenization. Their findings across 700 runs reveal that tokenization preserving reusable substructure—rather than vocabulary size or lexical similarity alone—determines transfer success, with transfer occurring in distinct stages from grammatical competence to masked lexical generalization.

This arXiv paper addresses a fundamental challenge in multilingual AI: understanding how language models transfer knowledge across languages when natural datasets confound multiple variables. By designing controlled synthetic languages with identical underlying structure but different surface forms, researchers eliminate noise that typically clouds cross-lingual analysis. This methodological innovation enables systematic investigation of which factors actually drive transfer versus which are mere correlates.

The research builds on growing recognition that tokenization significantly influences model behavior. Previous work observed that subword tokenization affects multilingual performance, but causality remained unclear. This study pins down the mechanism: vocabulary-level choices determine whether models learn decomposable, language-agnostic representations or language-specific atomic units. Smaller vocabularies force models to build words from shared fragments, enabling transfer; larger ones ossify language-specific forms, blocking it. This finding contradicts intuitions favoring large vocabularies for expressivity.

The staged transfer process—grammatical competence preceding lexical mastery—reveals model learning hierarchy. Models first internalize abstract structural patterns before tackling surface-level vocabulary mapping. This insight matters for practitioners deploying multilingual systems on low-resource languages. The correlation between tokenizer bridge strength and masked reachability provides an explainability framework, potentially enabling predictive assessment of transfer capability before deployment.

For AI researchers and practitioners, this work offers actionable design principles: optimize tokenizers for cross-lingual decomposability rather than coverage, expect staged competence development, and expect modest vocabulary sizes to improve zero-shot multilingual performance. The findings apply broadly to any multimodal or multi-domain transfer scenario where surface variation masks shared underlying structure.

• →Tokenization that preserves reusable substructure matters more than raw lexical similarity for cross-lingual transfer in language models
• →Smaller vocabularies often outperform larger ones by maintaining decomposable word structure shared across languages
• →Transfer emerges in stages with grammatical competence preceding masked lexical generalization
• →Tokenizer design fundamentally shapes whether models learn language-agnostic or language-specific representations
• →Bridge strength between tokenizer representations correlates strongly with masked word reachability