WhiFlash: Accelerating Speculative Decoding with Token-Level Cross-Paradigm Routing

WhiFlash introduces a novel speculative decoding method that combines autoregressive and diffusion-based drafting models through token-level routing, achieving up to 69.6% throughput improvements over existing approaches. The system uses lightweight controllers to dynamically switch between drafting paradigms based on per-token conditions, addressing a key bottleneck in LLM inference efficiency.

WhiFlash represents a meaningful advancement in LLM inference optimization, tackling the autoregressive bottleneck that constrains real-world deployment of large language models. The research identifies that current speculative decoding approaches use static routing strategies—selecting either autoregressive or diffusion-based drafting for entire sequences—which fails to capture significant performance variations within sequences. By implementing token-level routing with dynamic paradigm switching, WhiFlash exploits complementary strengths of fundamentally different architectural approaches.

The technical contribution extends beyond conceptual novelty. The system's cache-management optimizations—Lazy Catch-up and KV-only Prefill—reduce switching overhead to under 7% per-round latency, making high-frequency paradigm switching computationally viable. This engineering consideration transforms theoretical improvements into practical gains. The entropy-based and learned policy controllers provide tunable trade-offs between expected token gains and latency, enabling deployment flexibility across different workload requirements.

For the AI infrastructure sector, this development matters significantly. Inference efficiency directly impacts deployment costs and real-time application feasibility. The reported throughput gains—69.6% over EAGLE-3 and 37.3% over DFlash—suggest substantial practical improvements for production systems handling complex agentic workloads. This compounds with broader efforts to optimize transformer inference, including quantization and distillation techniques, creating cumulative efficiency gains.

Developers building AI applications benefit from faster inference latency and reduced computational requirements. As LLMs integrate into production systems requiring real-time responsiveness, WhiFlash-style optimizations become competitive differentiators. The research direction indicates future work may involve adaptive routing strategies and integration with emerging hardware accelerators.

• →WhiFlash achieves 69.6% throughput gains over autoregressive EAGLE-3 through dynamic cross-paradigm routing at token-level granularity.
• →Novel cache-management optimizations reduce switching overhead to below 7% of latency, enabling practical high-frequency paradigm selection.
• →The system uses either entropy-based or learned neural policies to route tokens between autoregressive and diffusion-based drafting models.
• →Token-level drafting accuracy varies significantly within sequences, revealing limitations of static routing approaches in prior speculative decoding methods.
• →Dynamic routing directly reduces inference latency and computational costs, with implications for production AI deployment efficiency.