#serving-infrastructure News & Analysis

4 articles tagged with #serving-infrastructure. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

4 articles

AIBullisharXiv – CS AI · Jun 107/10

🧠

K-Forcing: Joint Next-K-Token Decoding via Push-Forward Language Modeling

Researchers introduce K-Forcing, a novel language modeling approach that enables autoregressive models to generate multiple tokens simultaneously rather than sequentially, achieving 2.4-3.5x inference speedup. The technique distills existing AR models into a push-forward mapping trained via progressive self-forcing, maintaining compatibility with standard serving infrastructure while trading modest quality for significant computational efficiency gains critical for industrial-scale LLM deployment.

AINeutralarXiv – CS AI · May 277/10

🧠

Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks

Researchers have identified significant measurement bias in production LLM benchmarking tools, where single-process architectures and Python's Global Interpreter Lock artificially inflate latency metrics at scale. The study proposes a multi-process evaluation framework and a new normalized metric (NTPOT) to accurately measure LLM serving performance under production-level concurrency.

AIBullisharXiv – CS AI · Jun 236/10

🧠

Recency/Frequency Adaptive KV Caching for Large Language Model Serving

Researchers propose an adaptive key-value caching strategy for large language models that dynamically allocates cache space based on recency and frequency patterns, improving upon traditional LRU eviction policies. The approach demonstrates up to 10.8% improvement in cache hit rates and 12.6% reduction in time-to-first-token on synthetic workloads, with more modest gains on real-world conversation data.

AIBullisharXiv – CS AI · Jun 96/10

🧠

Harmonia: End-to-End RAG Serving Optimization

Harmonia is a new end-to-end RAG serving framework that optimizes the deployment and runtime performance of Retrieval-Augmented Generation pipelines. The system achieves 2.04x throughput improvements and reduces SLO violations by up to 78.4% through intelligent pipeline composition, heterogeneity-aware deployment, and dynamic load management.