BudgetDraft: Acceptance-Aware Multi-View Training for Sparse-KV Speculative Decoding

BudgetDraft is a new training method for sparse-KV speculative decoding that enables faster language model inference under memory constraints. By training drafters to handle multiple KV cache budgets simultaneously, the technique achieves up to 6.55x speedup on mid-to-long context inference while maintaining acceptance rates and reducing GPU memory usage.

BudgetDraft addresses a critical bottleneck in deploying large language models efficiently. Speculative decoding—where a smaller drafting model proposes tokens verified by a larger model—has emerged as a practical optimization technique. However, in resource-constrained environments, the sparse/full KV cache mismatch degrades performance as context length increases, limiting real-world applicability for common 4K-16K token contexts.

The innovation lies in BudgetDraft's acceptance-aware multi-view training paradigm. Rather than training a single drafter for one specific KV budget, the method exposes the model to multiple sampled budgets during training, forcing it to learn robust representations that work across sparsity levels. This contrasts with previous approaches requiring separate models or inference-time modifications. The technique combines acceptance-aware loss optimization with multi-view learning, producing a universally budget-robust drafter.

For infrastructure providers and LLM deployment teams, BudgetDraft directly impacts operational costs and inference latency. The reported speedups—6.55x at 4K context, declining to 2.10x at 16K—demonstrate meaningful throughput gains without architectural changes. Memory efficiency improvements reduce GPU resource requirements, enabling more concurrent requests per hardware unit. This particularly benefits edge deployments, mobile inference, and cost-sensitive cloud services.

The research validates performance across diverse benchmarks (PG-19, LongBench, LWM), suggesting broad applicability rather than benchmark-specific optimization. Industry adoption depends on integration into mainstream inference frameworks like vLLM or TensorRT. As context-aware applications proliferate, efficient inference methods become competitive differentiators for AI service providers.

• →BudgetDraft trains drafters on multiple KV budgets simultaneously, eliminating sparse/full cache mismatch degradation in mid-to-long context inference.
• →Achieves up to 6.55x end-to-end speedup at 4K context with minimal GPU memory overhead compared to autoregressive decoding.
• →Single model handles variable KV budgets without extra inference components, simplifying deployment versus prior multi-model approaches.
• →Performance validated across PG-19, LongBench, and LWM benchmarks demonstrates general applicability beyond single-domain optimization.
• →Method directly reduces operational costs for LLM inference services through improved throughput-to-memory efficiency ratios.