Theoretically Optimal Attention/FFN Ratios in Disaggregated LLM Serving

Researchers present an analytical framework for optimizing Attention/FFN provisioning ratios in disaggregated LLM serving architectures. The work provides closed-form rules and practical guidance for balancing memory-intensive attention computation with compute-intensive FFN operations, achieving predictions within 10% of simulation-optimal configurations.

This research addresses a critical infrastructure challenge in modern LLM deployment: how to efficiently allocate computational resources when separating attention and feedforward network components. Disaggregated architectures offer flexibility by allowing independent scaling of memory and compute, but require precise provisioning to avoid bottlenecks that waste resources through idle device time and processing delays.

The work emerges from growing industry recognition that monolithic LLM serving creates inefficiencies. Attention layers demand substantial memory for KV caches but consume less compute, while FFNs are compute-intensive but stateless. By disaggregating these components, operators can theoretically match resource allocation to actual demand. However, without principled guidance, provisioning becomes a costly optimization problem requiring expensive trial-and-error across diverse workloads.

The framework's key contribution lies in reducing this complex multi-variable problem to a single workload statistic (θ) that determines optimal provisioning. The closed-form mean-field rule identifies distinct regimes—attention-bounded, communication-bounded, and FFN-bounded—letting operators quickly determine appropriate A/F ratios for their specific workload characteristics. The Gaussian barrier refinement accounts for synchronization overhead across distributed workers, a practical consideration often overlooked in theoretical analyses.

For infrastructure providers and LLM hosting companies, this research directly impacts operational efficiency and cost optimization. More efficient provisioning reduces capital expenditure on unnecessary hardware and lowers operational costs through better resource utilization. The practical calibratable approach—validated against real traces—makes this immediately applicable rather than purely theoretical. As LLM inference costs become increasingly competitive, infrastructure innovations that improve efficiency gain substantial market value.

• →Disaggregated LLM architectures require precise Attention/FFN provisioning ratios to avoid costly bottlenecks and idle resources.
• →A single workload statistic (θ) governs optimal provisioning across different prefill-decode distributions, simplifying resource planning.
• →Closed-form rules identify attention-bounded, communication-bounded, and FFN-bounded regimes for targeted optimization.
• →Predictions match simulation-optimal configurations within 10%, validating the framework's practical applicability.
• →Infrastructure providers can immediately apply this framework to reduce capital and operational costs in LLM serving.