AIBullisharXiv – CS AI · Mar 47/103
🧠
Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving
Nightjar is a new adaptive speculative decoding framework for large language models that dynamically adjusts to system load conditions. It achieves 27.29% higher throughput and up to 20.18% lower latency by intelligently enabling or disabling speculation based on workload demands.