Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design

Researchers present a CPU-GPU hybrid system enabling local deployment of large Mixture-of-Experts models with cloud-level performance, achieving 1,800 tokens/s throughput and supporting 45K-token prompts within 30 seconds using consumer hardware. The breakthrough addresses critical gaps in local inference including latency, throughput, and concurrent workload handling without requiring quantization or model distillation.

This technical advancement addresses a significant bottleneck in AI infrastructure: the gap between cloud-scale large language model inference and consumer-grade local deployment. Currently, organizations relying on cloud providers face latency constraints, operational costs, and privacy concerns when processing large requests. The hybrid CPU-GPU approach using commodity hardware—dual-socket CPUs paired with consumer RTX GPUs—demonstrates that cloud-grade service-level objectives (SLOs) are achievable without enterprise-grade infrastructure.

The system's innovation centers on several engineering optimizations: stream-loading prefill (SLP) dramatically increases token throughput from typical 200-300 tokens/s to 1,200+ tokens/s; AVX-512 optimization enables native CPU FP8 inference, reducing latency by 4-5x; and disaggregation techniques allow concurrent handling of prefill and decode workloads without substantial latency penalties. These improvements enable processing 45K-token prompts within the critical 30-second threshold—essential for real-time applications.

For the AI infrastructure market, this represents a shift toward cost-effective local inference, reducing dependency on expensive cloud providers and enabling edge deployment scenarios. Organizations can now maintain inference capability on-premises using accessible hardware components. This democratization of MoE inference capability reshapes deployment economics and enables privacy-preserving applications where data cannot traverse cloud infrastructure.

Looking forward, the validation of consumer-grade hardware for enterprise-scale inference workloads may accelerate adoption of local deployment patterns. Further optimization of these techniques across different hardware configurations and MoE architectures will determine whether this represents a fundamental transition in inference infrastructure strategy.

• →Hybrid CPU-GPU systems achieve cloud-scale inference performance on consumer hardware without model quantization or distillation
• →Stream-loading prefill technology increases throughput to 1,800 tokens/s and handles 45K-token prompts within 30-second SLOs
• →AVX-512 optimizations enable 4-5x lower CPU latency for native FP8 inference on commodity processors
• →Prefill-decode disaggregation with shared weights sustains high concurrency with minimal latency penalties
• →Technical advances enable cost-effective, privacy-preserving local inference deployment for enterprise MoE models