Multilingual Fine-Tuning via Localized Gradient Conflict Resolution

Researchers introduce Bucket-Level MOO, a distributed framework that addresses negative interference when fine-tuning Large Language Models across multiple languages by reformulating the problem as multi-objective optimization. The method enables conflict-aware parameter updates without excessive communication overhead while theoretically guaranteeing Refined Pareto Stationarity, improving multilingual performance across four LLM architectures.

This research tackles a fundamental challenge in multilingual AI development: the tendency for fine-tuning on one language to degrade performance in others. When LLMs optimize for a single language during fine-tuning, gradient updates often create conflicting objectives across language-specific parameters, resulting in performance degradation on untrained languages. The Bucket-Level MOO framework reframes this as a multi-objective problem where multiple languages represent competing optimization targets that must be balanced rather than sacrificed sequentially.

The innovation lies in applying gradient-based multi-objective optimization locally within parameter buckets rather than globally across the entire model. This distributed approach circumvents the computational bottleneck of reconstructing and communicating full gradient vectors across a distributed system, making the solution practical at scale. Theoretically, the authors prove their method enforces Refined Pareto Stationarity, a stricter condition than traditional Pareto optimality, ensuring more principled trade-offs between language objectives.

The empirical results demonstrate the framework drives LLMs to develop distinct language-specific representational dimensions, enhancing separability and reducing interference. Testing across multiple base architectures strengthens the generalizability claim. For the AI industry, this addresses a critical pain point in building truly multilingual systems that maintain performance across supported languages. Companies and researchers developing global LLMs can reduce training iterations and fine-tuning cycles by preventing language-specific degradation. The work enables more efficient scaling of language coverage without architectural trade-offs, potentially accelerating deployment of equitable multilingual AI systems.

• →Bucket-Level MOO reformulates multilingual fine-tuning as multi-objective optimization to mitigate negative cross-language interference.
• →Local gradient-based MOO on parameter buckets eliminates prohibitive communication overhead while maintaining theoretical guarantees.
• →The method enforces Refined Pareto Stationarity, a stricter necessary condition than standard Pareto optimality.
• →Empirical validation across four LLM architectures shows improved performance on both seen and unseen languages.
• →Language-specific representational separability emerges naturally, enabling better multilingual capability preservation.