LiveCLKTBench: Towards Reliable Evaluation of Cross-Lingual Knowledge Transfer in Multilingual LLMs

Researchers introduce LiveCLKTBench, an automated benchmark for evaluating how well multilingual large language models transfer knowledge across languages, addressing the challenge of distinguishing genuine cross-lingual transfer from pre-training artifacts. Testing across five languages reveals that transfer effectiveness depends heavily on linguistic distance, model scale, and domain, with improvements plateauing in larger models.

LiveCLKTBench addresses a critical methodological gap in multilingual AI research by isolating genuine cross-lingual knowledge transfer from contamination effects in pre-training data. The benchmark's innovation lies in identifying time-sensitive, self-contained facts that likely weren't present during model training, then measuring how knowledge about these entities transfers across languages. This temporal filtering approach provides a more reliable foundation for understanding multilingual capabilities than previous evaluation methods.

The research builds on growing recognition that current multilingual LLMs exhibit uneven performance across language pairs and domains. As AI systems increasingly serve global users, understanding these transfer mechanisms becomes essential for predicting model behavior in low-resource and non-English contexts. Previous evaluation approaches couldn't cleanly separate genuine transfer from memorization, limiting insights into actual multilingual reasoning capabilities.

The findings have significant implications for AI development strategy. The observation that gains diminish with scale contradicts assumptions that simply training larger models solves multilingual challenges. The asymmetric transfer patterns across language directions suggest fundamental architectural or training factors that mere scale cannot overcome. Organizations developing multilingual systems must now confront that linguistic distance remains a persistent barrier regardless of model size, requiring targeted architectural innovations or training approaches.

Future work will likely focus on improving transfer mechanisms for distant language pairs and understanding the interplay between linguistic structure and knowledge retention. This benchmark enables systematic evaluation of proposed improvements, establishing a foundation for genuinely multilingual AI systems rather than English-centric models retrofitted for other languages.

• →LiveCLKTBench uses time-sensitive facts to isolate genuine cross-lingual transfer from pre-training contamination artifacts.
• →Cross-lingual knowledge transfer varies asymmetrically across language pairs and correlates strongly with linguistic distance.
• →Larger models improve cross-lingual transfer but show diminishing returns that plateau at scale.
• →Transfer effectiveness varies significantly across domains, requiring domain-specific evaluation strategies.
• →The benchmark provides a reliable methodology for future multilingual LLM research and development.