Redesign Mixture-of-Experts Routers with Manifold Power Iteration

Researchers propose Manifold Power Iteration (MPI), a novel router redesign method for Mixture-of-Experts models that aligns router rows with principal singular directions of associated experts. The approach uses a "Power-then-Retract" paradigm and demonstrates improved MoE model effectiveness across scales from 1B to 11B parameters.

This research addresses a fundamental architectural challenge in Mixture-of-Experts models, which have become increasingly important for scaling large language models efficiently. The router component determines which experts process each token, making its design critical to model performance. The authors identify that existing routers lack principled design methodology and propose aligning router representations with the dominant mathematical features of expert matrices.

The MPI approach builds on established linear algebra principles, specifically power iteration methods, to optimize router behavior. By performing power iteration on router weights followed by norm constraints, the method ensures both computational efficiency and numerical stability. This grounding in mathematical theory distinguishes it from heuristic improvements, providing theoretical guarantees about convergence toward optimal router-expert alignment.

For the AI development community, this work has practical implications for model efficiency and performance. Mixture-of-Experts architectures enable sparse computation, allowing models to scale parameters without proportionally increasing computational costs during inference. Better router design directly translates to more effective expert utilization, potentially improving model quality or reducing computational requirements. The empirical validation across multiple model scales from 1B to 11B parameters demonstrates scalability and practical applicability.

The research represents incremental but meaningful progress in neural network architecture optimization. As large language models continue scaling, such improvements in component design compound across systems. Organizations developing or deploying MoE-based models may benefit from these insights when optimizing for performance or efficiency. Future work likely builds on this foundation to further refine routing mechanisms or combine MPI with complementary architectural improvements.

• →MPI proposes aligning router rows with principal singular directions of expert matrices for more effective token-expert matching.
• →The method uses power iteration followed by norm constraints to ensure both mathematical optimality and computational stability.
• →Empirical validation across 1B to 11B parameter scales confirms improved MoE model performance with the new design.
• →Router optimization addresses a fundamental bottleneck in Mixture-of-Experts architectures used in modern large language models.
• →The approach grounds router design in linear algebra principles rather than heuristics, providing theoretical convergence guarantees.