DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks

Researchers introduce DynaMoE, a new Mixture-of-Experts framework that dynamically activates experts based on input complexity and uses adaptive capacity allocation across network layers. The system achieves superior parameter efficiency compared to static baselines and demonstrates that optimal expert scheduling strategies vary by task type and model scale.

• →DynaMoE removes fixed Top-K routing constraints by allowing variable numbers of experts to activate per token based on input complexity.
• →The framework implements six scheduling strategies for distributing expert capacity across network layers including descending, ascending, pyramid, and wave patterns.
• →Optimal expert schedules are task-dependent: descending schedules work best for image classification while language modeling requires different strategies by model size.
• →Dynamic routing reduces gradient variance during training, leading to improved convergence stability.
• →Extensive testing across MNIST, Fashion-MNIST, CIFAR-10, and language modeling tasks validates the approach's effectiveness.