Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution

Researchers demonstrate that Mixture of Experts (MoE) models contain substantial underutilized sparsity within individual experts that can be exploited without modifying model parameters. By implementing intra-expert activation sparsity in vLLM, they achieve up to 2.5x speedup in MoE layer execution, offering a practical optimization path for efficient large language model deployment.

This research addresses a critical efficiency challenge in scaling large language models. While MoE architectures have enabled parameter-efficient training through sparse expert selection, the field has struggled with fundamental limitations like expert collapse and load imbalancing. The discovery that existing pre-trained models already contain 90% potential sparsity within individual experts represents a significant overlooked optimization opportunity.

The work builds on the foundation of MoE's success as the architecture of choice for state-of-the-art LLMs, where only a subset of experts activate per token. By analyzing eight models ranging from 1B to 400B parameters, the researchers validate that intra-expert sparsity is a consistent, reliable phenomenon across model scales. This consistency suggests the pattern emerges naturally during training rather than requiring specialized techniques.

The practical implementation in vLLM demonstrates immediate real-world applicability. Achieving 1.2x end-to-end speedup alongside 2.5x MoE-specific improvements means inference costs decrease without sacrificing model quality or requiring retraining. This matters substantially for organizations deploying massive language models, where inference costs often exceed training expenses. The approach complements existing optimization efforts rather than replacing them, offering compounding benefits.

Moving forward, this research opens questions about whether similar sparsity patterns exist in other dense layers and whether deliberately training models to maximize intra-expert sparsity could yield additional gains. The findings suggest the current efficiency ceiling for MoE inference is substantially higher than previously assumed.

• →Existing pre-trained MoE models contain up to 90% latent sparsity within individual experts without any architecture modifications.
• →Implementing intra-expert sparsity in vLLM achieves 2.5x speedup specifically in MoE layers and 1.2x overall inference speedup.
• →The optimization requires no model retraining or parameter changes, making immediate deployment feasible across existing models.
• →Intra-expert sparsity provides a complementary optimization dimension to address MoE's training challenges like expert collapse.
• →The approach is validated across eight models from 1B to 400B parameters, demonstrating scalability across model sizes.