One Model to Translate Them All? A Journey to Mount Doom for Multilingual Model Merging

Researchers studied weight-space model merging for multilingual machine translation and found it significantly degrades performance when target languages differ. Analysis reveals that fine-tuning redistributes rather than sharpens language selectivity in neural networks, increasing representational divergence in higher layers that govern text generation.

• →Weight-space model merging fails in multilingual machine translation contexts, especially when target languages are different.
• →Language-specific neurons concentrate in embedding layers and upper transformer blocks while intermediate layers remain shared across languages.
• →Fine-tuning redistributes language selectivity rather than making it more precise, reducing compatibility with standard merging methods.
• →Neurons for supervised languages become less exclusive while unsupervised language neurons grow more isolated during fine-tuning.
• →The research provides explanation for why standard model merging assumptions don't work in multilingual scenarios.