FMplex: Model Virtualization for Serving Extensible Foundation Models

FMplex is a new model-serving system that enables multiple downstream tasks to share a single foundation model backbone through virtualization, reducing memory waste and computational costs. The system achieves up to 80% latency reduction compared to traditional spatial partitioning approaches while enabling clusters to host 6x more tasks simultaneously.

FMplex addresses a critical inefficiency in current foundation model deployment architectures. As organizations increasingly customize FMs for specific downstream applications, traditional approaches deploy separate model instances for each task, resulting in substantial redundancy. The shared backbone remains static across tasks, yet replicating it entirely wastes accelerator memory and prevents efficient batching across independent workloads. This architectural limitation becomes more pronounced as FM inference costs grow and organizations seek to maximize hardware utilization.

The system's innovation centers on treating FM backbones as virtualization substrates, similar to how operating systems manage computational resources. Each task receives a virtual FM that maintains logical independence while sharing the underlying physical model. The accompanying batch-aware fair-queueing scheduler ensures equitable resource distribution while capitalizing on inter- and intra-task batching opportunities. This technical approach mirrors advances in container orchestration and resource virtualization from cloud infrastructure, applied specifically to the ML serving domain.

For AI infrastructure providers and enterprises running production FM deployments, FMplex represents meaningful operational efficiency gains. The 33.3% latency improvement over best-effort co-location translates to faster inference and reduced serving costs, while hosting 6x more tasks per cluster reduces capital expenditure requirements. The breadth of evaluation across 7 FM backbones and 92 downstream tasks suggests broad applicability rather than narrow optimization for specific architectures.

Deployment of virtualization-based serving systems like FMplex will likely shape how organizations architect their ML infrastructure. Success could accelerate consolidation of serving infrastructure, reducing per-inference costs and enabling smaller organizations to run multiple specialized models efficiently.

• →FMplex reduces latency by up to 80% compared to spatial partitioning through shared foundation model virtualization
• →The system enables clusters to host 6x more tasks simultaneously while maintaining task-level isolation and independence
• →Virtual foundation models preserve task-specific customizations while sharing heavyweight backbones across multiple downstream applications
• →Batch-aware fair-queueing scheduler combines weighted task sharing with both inter-task and intra-task batching optimization
• →Evaluation across 16 FM variants and 92 tasks demonstrates broad applicability beyond narrow use cases