Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding

Researchers propose Hybrid Verified Decoding, a technique that improves LLM inference speed by intelligently choosing between cache-based and model-based token drafting methods. The approach predicts draft acceptance rates before verification, achieving 2.73x average speedup on agentic workflows and outperforming existing methods like EAGLE3.

Hybrid Verified Decoding addresses a fundamental computational bottleneck in large language model inference. Traditional autoregressive decoding generates one token at a time, requiring full model passes for each output. Speculative decoding mitigates this by drafting multiple tokens cheaply and verifying them together, but effectiveness depends on draft acceptance rates—rejected tokens waste computation.

The research builds on growing recognition that different workloads exhibit different bottlenecks. Structured and agentic tasks create repetitive patterns amenable to cache-based drafting, where previously computed token sequences can be reused without model calls. However, draft quality varies across generation steps, making static draft selection suboptimal. This work introduces payoff prediction, estimating whether a cached draft will be accepted before expensive verification occurs.

For the AI infrastructure industry, this advancement matters because inference cost drives deployment economics. The 2.73x speedup on agentic workflows—a rapidly growing application category—directly reduces operational expenses for companies running LLM services. The finding that high-payoff cache drafts concentrate in specific regions of the draft space suggests further optimization opportunities through learned selection policies.

The research also reveals how prompt structure itself influences acceleration opportunities, indicating that inference optimization and prompt engineering interact meaningfully. As LLM deployment scales, techniques that adaptively allocate verification budget based on workload characteristics become increasingly valuable. The emphasis on runtime draft selection as a promising direction suggests this remains an unsolved frontier for the field, with potential for even greater efficiency gains as methods mature.

• →Hybrid Verified Decoding achieves 2.73x average speedup on agentic workflows by predicting draft acceptance before verification
• →The method intelligently selects between cache-based and model-based drafters, improving upon fixed-strategy approaches like EAGLE3
• →High-payoff cache drafts concentrate in specific regions, enabling targeted optimization of speculative decoding
• →Prompt structure directly influences cache opportunities, linking inference optimization to prompt design patterns
• →Payoff-guided draft selection reduces sequential decoding work, establishing runtime selection as a key research direction