#distributed-serving News & Analysis

2 articles tagged with #distributed-serving. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

2 articles

AIBullisharXiv – CS AI · May 287/10

🧠

How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving

Researchers present a systematic study of Attention-FFN Disaggregation (AFD), a technique that separates attention and expert layers across different GPU groups to optimize inference serving for Mixture-of-Experts language models. The framework demonstrates that AFD enables 4k tokens/s throughput on DeepSeek-V3.2 under strict latency constraints where traditional disaggregation approaches fail, providing design principles for scaling LLM infrastructure.

AINeutralarXiv – CS AI · Jun 96/10

🧠

Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patching

Researchers propose Semantic Cache Distillation (SCD), a technical framework that significantly reduces communication overhead in large language model inference by replacing raw Key-Value cache transmission with compact semantic codes. The method achieves up to 2.65x speedup in time-to-first-token while maintaining generation quality within 5% of baseline performance, addressing a critical bottleneck in disaggregated LLM serving architectures.