ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference

Researchers introduce ReMoE, a router fine-tuning framework that optimizes Mixture-of-Experts language models for memory-constrained inference by increasing expert reuse and reducing storage I/O overhead. The approach improves expert reuse by 26% while maintaining performance, delivering up to 1.99× decode speedup on edge devices.

ReMoE addresses a critical bottleneck in deploying large Mixture-of-Experts models on resource-limited hardware. Modern MoE architectures like DeepSeek and Qwen sparsely activate experts to reduce computation, but memory constraints force most experts into slow external storage. When the router requests uncached experts, expensive I/O operations degrade throughput substantially. ReMoE solves this by fine-tuning the router to favor recently accessed experts, creating temporal stability that aligns routing decisions with cache locality—a simple yet effective approach requiring no additional inference-time computation.

This work emerges from the broader trend of democratizing large language model inference. As model sizes exceed GPU memory, techniques like expert offloading become practical necessities rather than optimizations. ReMoE's 26% improvement in expert reuse translates to tangible real-world gains: 8.4% throughput improvement on GPU-CPU offloading scenarios and 43-50% latency reduction on edge accelerators like Jetson Orin NX. These metrics matter because they expand the deployment envelope for sophisticated models to consumer devices and inference-constrained environments.

For developers and researchers, ReMoE offers immediately applicable optimization without architectural changes or additional training overhead—only router fine-tuning. The open-source release amplifies impact by enabling rapid adoption. The approach fundamentally validates that routing patterns in sparse models can be shaped toward hardware constraints, opening avenues for hardware-aware model design. As MoE architectures dominate next-generation LLMs, such memory-efficient inference techniques become increasingly critical for practical deployment.

• →ReMoE boosts expert reuse by 26% through router fine-tuning, reducing expert fetches from slow storage without computational overhead.
• →Real-system tests show 8.4% GPU-CPU throughput gains and up to 1.99× decode speedup on edge devices like Jetson Orin NX.
• →The framework biases routers toward recently selected experts to improve cache locality alignment and temporal stability.
• →No additional inference-time computation required, making deployment straightforward across diverse hardware platforms.
• →Open-source checkpoints enable rapid adoption for optimizing MoE inference in memory-constrained environments.