Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformers

Researchers have developed Variational Mixture-of-Experts Routing (VMoER), a Bayesian framework that enables uncertainty quantification in large-scale AI models while adding less than 1% computational overhead. The method improves routing stability by 38%, reduces calibration error by 94%, and increases out-of-distribution detection by 12%.

• →VMoER introduces Bayesian uncertainty quantification to Mixture-of-Experts layers in foundation models with minimal computational cost.
• →The framework improves routing stability under noise by 38% and reduces calibration error by 94%.
• →Out-of-distribution detection capability increases by 12% while adding less than 1% additional computational overhead.
• →The approach confines Bayesian inference to the expert-selection stage rather than the entire model.
• →This offers a scalable path toward more robust and uncertainty-aware foundation models at trillion-parameter scale.