Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Serving

Researchers challenge the necessity of expensive high-bandwidth networks for Mixture-of-Experts LLM serving, demonstrating that lower-cost switchless topologies deliver 20.6-56.2% better cost-effectiveness than industry-standard scale-up architectures. The analysis reveals current network infrastructure is over-provisioned, with implications for data center economics and AI deployment efficiency.

The infrastructure economics of large language model serving are undergoing significant reevaluation. This research introduces rigorous cross-layer analysis demonstrating that the industry's heavy investment in expensive scale-up networks may be economically irrational. By comparing four distinct XPU topologies across diverse serving scenarios, the researchers establish that switchless architectures—particularly 3D full-mesh configurations—achieve superior performance-per-dollar metrics.

The findings emerge from a critical inefficiency in current AI infrastructure deployment. As Mixture-of-Experts models shift serving from single-node to cluster-scale workloads, communication overhead became a primary bottleneck, prompting costly hardware investments. However, this research quantifies what many practitioners suspected: marginal performance gains don't justify the premium pricing of scale-up networks.

For cloud providers and data center operators, these insights carry substantial financial consequences. Adopting cost-optimized topologies could reduce capital expenditure on networking hardware while maintaining or improving throughput. This democratizes advanced LLM serving, enabling smaller operators to compete economically with hyperscalers investing in expensive infrastructure. The discovery that current link bandwidths are over-provisioned suggests further optimization opportunities through intelligent link dimensioning.

Looking forward, the analysis projects that emerging GPU generations will maintain or amplify these advantages for switchless networks. This suggests a structural shift in how organizations should architect inference clusters. The implications extend beyond cost savings—they signal a maturation of the AI infrastructure ecosystem where empirical analysis trumps vendor recommendations, potentially redirecting billions in capital allocation away from premium networking vendors toward alternative infrastructure investments.

• →3D full-mesh switchless topologies achieve 20.6-56.2% better cost-effectiveness than traditional scale-up networks for MoE LLM serving
• →Current industry-standard scale-up networks are over-provisioned, with bandwidth reductions improving throughput-per-cost by up to 27%
• →The cost-performance advantage of switchless architectures is expected to persist across future GPU generations
• →Lower-cost network topologies eliminate the necessity for expensive high-bandwidth infrastructure investments in AI clusters
• →The findings challenge existing infrastructure procurement strategies at hyperscalers and enterprise AI deployments