SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignment

Researchers introduce SARA, a framework that improves multilingual performance in Mixture-of-Experts language models by aligning routing patterns between low-resource and high-resource languages. The method uses semantic anchoring and Jensen-Shannon divergence constraints to enable better expert sharing across languages, demonstrating measurable improvements on benchmark tests.

SARA addresses a fundamental challenge in multilingual AI systems: the routing divergence problem in sparse Mixture-of-Experts architectures. When language models use MoE designs—which activate only specialized subsets of parameters for efficiency—low-resource languages typically route to different expert modules than high-resource languages, preventing beneficial knowledge transfer. This fragmentation reduces the effectiveness of multilingual models despite their architectural advantages in scalability.

The research builds on growing recognition that language model performance inequality across linguistic groups requires targeted technical solutions beyond simple data augmentation. Previous approaches focused on output-level distillation, treating the symptom rather than the underlying routing mechanism. SARA innovates by working at the mechanistic level, explicitly constraining how tokens from different languages map to expert modules through symmetric Jensen-Shannon divergence—a statistical measure of distribution similarity.

The experimental validation across two LLM architectures (Qwen3 and Phi-3.5) and multiple low-resource languages demonstrates consistent, if modest, improvements (+0.8% to +1.2% on standard benchmarks). This consistency suggests the approach addresses a real bottleneck rather than achieving gains through noise or overfitting. For developers deploying multilingual systems, this offers a practical technique to enhance performance in underserved languages without proportional increases in computational cost.

Looking forward, the significance lies in establishing mechanistic alignment as a viable optimization pathway for sparse models. As MoE architectures become increasingly standard for scaling, methods addressing their cross-lingual limitations gain importance. The framework's generalizability across different model families hints at potential broader applicability to other sparse architectures.

• →SARA aligns expert routing between low-resource and high-resource languages using Jensen-Shannon divergence constraints to improve cross-lingual knowledge transfer.
• →The framework operates at the routing mechanism level rather than output logits, addressing the fundamental divergence problem in multilingual MoE models.
• →Experimental results show consistent improvements of 0.8-1.2% on standard multilingual benchmarks across different LLM architectures.
• →The method enables efficient multilingual scaling without requiring additional training data for low-resource languages.
• →SARA's success with multiple models suggests mechanistic alignment is a generalizable optimization approach for sparse architectures.