Researchers demonstrate that exclusive batching (EB) can outperform the industry-standard mixed batching (MB) approach for LLM inference on bandwidth-constrained GPUs, with performance crossover dependent on hardware specifications and workload composition. A new hybrid scheduler (EB+) dynamically switches between strategies to optimize throughput across varying traffic conditions.
This research addresses a fundamental optimization problem in LLM inference that has direct implications for AI infrastructure efficiency. Mixed batching has dominated inference scheduling because it interleaves prefill and decode operations to maximize resource utilization, but the study reveals this approach creates interference costs that become prohibitive under specific conditions. The findings expose a critical gap between theoretical efficiency and practical performance on different hardware tiers.
The hardware-dependent nature of the optimal strategy represents a significant shift in how inference systems should be designed. On high-bandwidth GPUs like the H200, mixed batching remains superior even with heavy decode loads, but on constrained bandwidth hardware, the crossover point occurs at just 20% decode tokens. This variability means a one-size-fits-all scheduling approach systematically underperforms across heterogeneous infrastructure environments.
For the AI infrastructure industry, this research directly impacts cloud provider costs and latency performance. Inference platforms serving diverse workloads can improve throughput by 36-41% by dynamically adapting scheduling strategies. This efficiency gain translates to reduced operational costs and improved user experience without hardware upgrades. The EB+ scheduler's ability to adapt online without manual tuning makes deployment practical at scale.
Looking ahead, this work will likely influence how inference engines implement scheduling logic in production systems. The closed-form mathematical conditions enable straightforward implementation in existing frameworks. As inference becomes increasingly cost-sensitive and latency-critical for LLM applications, adaptive scheduling strategies could become a competitive differentiator for inference service providers.
- →Mixed batching's optimality depends critically on GPU memory bandwidth, not just general compute capacity.
- →Exclusive batching can deliver 41.9% higher throughput on bandwidth-constrained hardware compared to fixed mixed batching.
- →The mathematical crossover point between exclusive and mixed batching enables automated, adaptive scheduling decisions.
- →Dynamic scheduler EB+ maintains near-optimal performance across varying traffic distributions without manual configuration.
- →Inference efficiency gains of this magnitude can significantly reduce operational costs for large-scale LLM deployment.