ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge--Cloud Speculative LLM Serving

ConfigSpec introduces a profiling-based framework for optimizing distributed LLM inference across edge-cloud systems using speculative decoding. The research reveals that no single configuration can simultaneously optimize throughput, cost efficiency, and energy efficiency—requiring dynamic, device-aware configuration selection rather than fixed deployments.

ConfigSpec addresses a critical infrastructure challenge in modern LLM deployment: the explosion of configuration variables when distributing inference across heterogeneous edge and cloud environments. Speculative decoding, which separates lightweight token drafting from heavy verification workloads, has emerged as a promising optimization technique, but practical implementation requires navigating dozens of interdependent parameters including draft model size, quantization levels, speculation depth, and device hardware profiles.

The research builds on the growing recognition that LLM inference is becoming increasingly distributed. As cloud costs rise and latency-sensitive applications proliferate, edge deployment has gained traction—yet edge devices vary dramatically in computational capacity, memory, and power constraints. Prior systems achieved performance gains but lacked systematic frameworks for selecting optimal configurations across this heterogeneous landscape.

ConfigSpec's core contribution lies in exposing fundamental trade-offs between optimization objectives. The finding that goodput peaks with small, fast draft models while cost and energy efficiency converge around K=2 speculation length reveals that infrastructure operators cannot achieve one-size-fits-all solutions. This fragmentation across objectives has direct implications for MLOps platforms and inference services: systems must implement runtime profiling and dynamic configuration switching rather than deploying static models.

For the broader AI infrastructure market, ConfigSpec validates the business case for configurable inference platforms and profiling-as-a-service offerings. Organizations running LLM workloads face pressure to reduce both latency and operational costs simultaneously—this research demonstrates those goals often conflict at the configuration level. The work suggests future inference platforms will need sophisticated profiling capabilities and multi-objective optimization as competitive differentiators.

• →No single configuration optimizes throughput, cost, and energy efficiency simultaneously in distributed speculative LLM serving
• →Smallest draft models maximize throughput at device-dependent speculation depths of K*=2-10 tokens
• →Optimal speculation length converges to K=2 for cost and energy efficiency due to bonus-token effects
• →Profiling-based dynamic configuration selection is essential for practical edge-cloud LLM deployment
• →Cost efficiency favors larger drafters while energy efficiency favors smaller drafters, creating structural conflicts