Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design

Researchers have developed a new scaling law for Mixture-of-Experts (MoE) models that optimizes compute allocation between expert and attention layers. The study extends the Chinchilla scaling law by introducing an optimal ratio formula that follows a power-law relationship with total compute and model sparsity.

• →A novel extension of neural scaling laws specifically addresses compute allocation in Mixture-of-Experts models.
• →The optimal expert-attention compute ratio follows a power-law relationship with total compute budget and varies with model sparsity.
• →The research provides an explicit formula for determining the optimal ratio, enabling precise architectural control.
• →The findings generalize the Chinchilla scaling law by incorporating architectural parameters beyond just model size and training data.
• →The framework offers practical guidelines for designing more efficient MoE models within fixed compute budgets.