When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference

A research study reveals that NPUs (Neural Processing Units) on mobile devices don't consistently accelerate LLM inference as expected, with CPUs outperforming NPUs on compute-intensive prefill operations and NPUs providing only marginal speedups on memory-bound decode stages. The findings challenge assumptions about heterogeneous mobile computing and suggest current NPU designs require architectural improvements for on-device AI workloads.

Mobile AI deployment has increasingly assumed that offloading language model computation to specialized NPUs delivers automatic performance gains, but this arXiv study presents empirical evidence contradicting that premise. Researchers conducted the first stage-level analysis of LLM inference on heterogeneous CPU-NPU systems, discovering fundamental misalignments between NPU architecture and LLM execution patterns. The prefill stage, which dominates compute operations, executes 1.6x faster on CPUs, while the memory-bound decode stage sees NPU acceleration capped at 1.05-1.2x due to bandwidth limitations. This performance reversal stems from NPU design optimization for specific neural network topologies rather than the irregular computation patterns LLMs exhibit.

The research introduces critical context about deployment overhead often ignored in theoretical benchmarks. Scheduling complexity and cross-backend fallback mechanisms impose tangible costs that erode NPU benefits in practice. Energy measurements reveal an unexpected penalty: increased NPU offloading correlates with up to 51% higher energy consumption, contradicting marketing claims about mobile efficiency gains. This finding carries significant implications for device manufacturers pursuing on-device AI features and developers making architecture decisions.

For the semiconductor and mobile computing industries, these results suggest the current generation of NPU designs require fundamental rethinking. The study provides actionable design guidelines for NPU architects, indicating that future improvements must address irregular memory access patterns and reduce cross-backend communication overhead. Developers currently prioritizing NPU offloading strategies may need to reconsider CPU-bound execution paths for latency-critical applications, while vendors marketing NPU capabilities should adjust performance expectations for LLM workloads.

• →CPUs outperform NPUs on compute-intensive prefill stages by up to 1.6x, contradicting assumptions about specialized processor benefits
• →NPU acceleration in memory-bound decode stages delivers marginal 1.05-1.2x speedups due to bandwidth constraints rather than compute limitations
• →Scheduling overhead and cross-backend fallback mechanisms significantly reduce practical NPU benefits in real-world deployments
• →Increased NPU offloading correlates with up to 51% higher energy consumption, eliminating expected efficiency gains
• →Current NPU architectures require redesign to effectively support LLM inference patterns rather than traditional neural network topologies