Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study

A new empirical study challenges the assumption that Mixture-of-Experts language models deliver practical inference speed advantages on consumer and edge hardware, finding that MoE models underperform comparable dense models due to bandwidth constraints and memory overhead rather than computational limitations.

The research addresses a critical gap between theoretical efficiency and real-world performance in machine learning inference. While MoE architectures promise reduced compute by activating only a subset of expert parameters per token, this study reveals the promise doesn't materialize consistently across hardware. Testing OLMoE-1B-7B against dense baselines shows a 10% performance deficit on Apple M2 Pro and a severe 31% deficit on NVIDIA Jetson Orin Nano edge hardware, with energy consumption doubling on the constrained device. This matters significantly for developers deploying models to resource-limited environments, a growing use case as AI applications proliferate at the edge. The root cause isn't routing overhead, which consumed under 9% of MoE-block compute, but rather total parameter memory footprint and KV-cache pressure. On bandwidth-bound hardware, the system can't exploit sparse activation benefits because memory access patterns, not raw computation, become the bottleneck. For the AI infrastructure industry, this suggests MoE optimization strategies must account for hardware characteristics rather than assuming parameter efficiency translates universally. The findings indicate that edge deployment decisions should weigh total model size against available bandwidth rather than relying on active-parameter counts. Future optimization efforts should focus on reducing memory overhead and improving cache efficiency for sparse models. The limited scope—one MoE model and two devices—means these conclusions warrant validation across broader architectures and hardware platforms before drawing universal conclusions.

• →MoE models underperform dense models on edge hardware despite theoretical FLOP advantages, running 31% slower on Jetson Orin with 2.1× energy overhead per token.
• →Bandwidth constraints, not routing complexity, drive poor MoE performance on resource-limited devices, with memory footprint mattering more than active parameters.
• →The 10% deficit on M2 Pro versus 31% on Jetson Orin demonstrates device-dependent results where sparse activation benefits depend heavily on hardware characteristics.
• →Routing operations consume less than 9% of MoE-block compute time, indicating memory access patterns and KV-cache pressure are the primary performance limiters.
• →Developers deploying to edge hardware should prioritize total model size and memory efficiency over active-parameter counts when selecting between MoE and dense architectures.