Researchers introduce TAPER, an admission controller for managing parallel branch execution in LLM serving systems. The system dynamically regulates how many concurrent decoding branches are allowed per request step, balancing throughput gains against degradation to co-batched requests, achieving 1.77x improvement in goodput over conservative baselines.
LLM serving optimization has emerged as a critical infrastructure challenge as these models scale. Traditional approaches to parallel decoding either greedily accept all branches regardless of system load or impose fixed caps that leave performance on the table. TAPER addresses this fundamental tension by treating branch admission as a dynamic resource allocation problem, where each scheduling decision accounts for the actual computational slack available in the current batch configuration.
The technical innovation centers on the concept of 'branch externality'—the latency imposed on other requests when new branches are admitted. By predicting this cost and only admitting branches when slack exists to absorb them, TAPER achieves better overall system utilization. The approach is practical because branches share a request's KV cache prefix, enabling width adjustments without expensive memory reclamation operations.
For the AI infrastructure market, this work demonstrates that naive parallelism strategies waste system capacity. Companies operating large-scale LLM inference services—cloud providers, AI startups, and internal deployments—stand to benefit from methods that improve goodput by 48-77% while maintaining SLO compliance. The 95%+ SLO attainment rate suggests production viability.
The research validates an emerging principle: dynamic per-step scheduling outperforms static policies in heterogeneous serving workloads. Future LLM serving stacks will likely incorporate adaptive branch admission as a standard component. The work also opens questions about extending similar principles to other serving optimization dimensions like speculative decoding and multi-model batching.
- →TAPER improves LLM serving goodput by 1.77x compared to conservative fixed-cap branch admission approaches
- →Dynamic per-step regulation of parallel branches prevents performance degradation to co-batched requests while capturing parallelism benefits
- →The branch externality concept quantifies latency costs imposed on other requests, enabling principled admission decisions
- →Shared KV cache architecture allows efficient width adjustments without memory reclamation overhead
- →System maintains over 95% SLO attainment while achieving substantial throughput gains on production-scale models