88 articles tagged with #llm-inference. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
OpenAI and Broadcom have jointly announced a custom chip specifically designed for large-scale language model inference, intensifying competition in AI silicon development. This move reflects the industry's urgent need for specialized hardware to handle growing demand for LLM deployment at scale.
OpenAI and Broadcom have jointly developed Jalapeño, a custom AI chip specifically optimized for large language model inference operations. The chip aims to enhance performance and energy efficiency while improving scalability for AI systems, representing a strategic move by OpenAI to reduce dependency on third-party semiconductor providers.
Researchers introduce GDP-RAG, a novel retrieval-augmented generation framework that improves multi-hop question answering by focusing computation only on information gaps rather than over-generating reasoning steps. The system achieves 60.63% accuracy on benchmark datasets while reducing computational costs by 22-68% compared to existing approaches.
Researchers propose Geometry-Aware Online Scheduling, introducing the Smallest Volume First (SVF) algorithm to optimize LLM inference by accounting for dynamic memory footprint of Key-Value caches. The approach improves upon traditional time-centric scheduling heuristics, achieving significant reductions in latency and throughput gains when integrated into vLLM.
Researchers develop a delay-adaptive algorithm for optimizing speculative decoding in distributed LLM inference across edge-cloud systems. The study proves optimal draft length follows a finite threshold policy and introduces UCB-SpecStop, an online control algorithm that reduces per-token latency by up to 22.4% compared to existing methods while adapting to varying network conditions.
Researchers propose VIA-SD, a multi-tier verification framework for speculative decoding that uses a lightweight slim-verifier to handle medium-confidence tokens instead of always invoking full model verification. The approach reduces rejection rates by 10-22% and achieves 10-20% speedup improvements over existing speculative decoding methods while maintaining compatibility with current frameworks.
TileFuse is a new kernel library that enables efficient quantized large language model inference on AMD's XDNA2 NPUs by supporting industry-standard quantization formats like AWQ directly, rather than requiring model reshaping. The technology delivers up to 2x improvements in latency and energy efficiency on edge devices, making practical LLM deployment on consumer hardware substantially more viable.
Researchers propose CLP (Collocation-Length Predictor), a lightweight neural architecture that improves multi-token prediction inference for large language models by eliminating competition between prediction heads and backbone models. The method achieves 1.20x-1.29x speedup on smaller models with zero quality degradation, significantly outperforming existing approaches that suffer from repetitive outputs.
WhiFlash introduces a novel speculative decoding method that combines autoregressive and diffusion-based drafting models through token-level routing, achieving up to 69.6% throughput improvements over existing approaches. The system uses lightweight controllers to dynamically switch between drafting paradigms based on per-token conditions, addressing a key bottleneck in LLM inference efficiency.
Researchers introduce APEX4, a pure INT4 inference system that addresses the long-standing challenge of W4A4 quantization in large language models by adapting compute strategies based on GPU architecture. The system achieves up to 2.09× speedup on consumer GPUs while maintaining quality within 0.63 perplexity points of FP16 baselines, making efficient LLM inference more practical across diverse hardware platforms.
Researchers propose Reset-and-Discard (ReD), a novel querying method that improves large language model inference efficiency by optimizing the coverage@cost metric—the number of unique questions answered within a fixed budget. The technique reduces computational attempts, tokens, and financial costs needed to achieve desired performance levels across coding, math, and reasoning tasks.
Researchers introduce SIFT, a novel optimization technique for Retrieval-Augmented Generation (RAG) systems that exploits attention patterns to accelerate LLM prefill computation. By storing only compact bit vectors of high-attention locations rather than full KV tensors, SIFT achieves 1.71x faster time-to-first-token while reducing storage by up to 24,000x and maintaining accuracy within 1% of standard methods.
Researchers introduce STAR-KV, an adaptive compression framework that reduces KV cache memory requirements in large language models by up to 75% through low-rank projections and intelligent rank selection. The technique achieves up to 20x compression when combined with quantization and delivers significant speedups in attention computation, addressing a critical bottleneck in LLM inference efficiency.
ThinkBooster is a unified framework that standardizes test-time compute scaling for large language models, providing a modular library, benchmarking suite, and production-ready API for improving LLM reasoning efficiency during inference. The framework enables developers to evaluate and deploy adaptive reasoning strategies with transparent performance-compute trade-offs across mathematical and coding tasks.
RedKnot is a new KV cache management system for large language models that optimizes memory efficiency by treating cache differently across attention heads rather than as a uniform block. This head-aware approach enables better resource utilization, higher serving concurrency, and improved scalability without requiring model retraining.
Vortex is a new system that simplifies the development and deployment of sparse attention algorithms for large language models, enabling researchers and AI agents to rapidly prototype and evaluate efficiency improvements. The platform demonstrates substantial real-world performance gains, with optimized algorithms achieving up to 3.46× higher throughput than full attention while maintaining accuracy, and successfully extending sparse attention to emerging model architectures.
QCFuse introduces a compressed-view query-aware selector for retrieval-augmented generation (RAG) systems that accelerates LLM serving by intelligently reusing cached key-value computations. The technique achieves 1.7x speedup over full prefill and 1.5x over existing baselines while maintaining full-prefill quality, addressing a critical bottleneck in RAG deployment.
Researchers propose Cross-Layer Sparse Attention (CLSA), a novel architecture that optimizes long-context LLM inference by sharing both key-value caches and routing indices across decoder layers. The method achieves up to 7.6x decoding speedup and 17.1x throughput improvement at 128K context while maintaining accuracy, addressing the efficiency-quality tradeoff that has constrained existing sparse attention approaches.
Researchers introduce Ekka, an automated diagnostic system that identifies root causes of silent errors in large language model serving frameworks by comparing execution states between target and reference implementations. The system achieves 80% pass@1 accuracy and has already discovered 4 new bugs in production serving frameworks, addressing a critical reliability challenge in LLM deployment.
Researchers introduce SSSD, a training-free method for accelerating Large Language Model inference that reduces latency by up to 2.9x through n-gram matching and hardware-aware speculation. The approach matches performance of existing trained methods while eliminating deployment complexity, data preparation, and maintenance overhead.
Researchers propose CLEAR, an economic optimization framework for allocating computational budgets during LLM inference by modeling resource allocation as a constrained optimization problem. The approach uses a global shadow price mechanism to redistribute tokens from queries unlikely to succeed to those near performance thresholds, achieving up to 3x accuracy improvements in resource-constrained environments.
Researchers introduce Lodestar, a machine learning-based request routing system that dynamically assigns large language model inference tasks to GPU instances in distributed clusters. The system achieves up to 4.38x improvements in latency metrics compared to existing heuristics by continuously learning optimal routing strategies in real-time.
Researchers introduce DyLLM, a training-free inference framework that accelerates diffusion language model decoding by up to 9.6x by selectively computing only salient tokens rather than processing entire sequences at each step. The approach identifies important tokens through attention context similarity and reuses cached activations for stable tokens, maintaining baseline accuracy across benchmarks.
SENSE is a new retrieval-based speculative decoding method that accelerates LLM inference by using semantic embeddings instead of lexical matching to retrieve candidate tokens. The approach achieves up to 3.26x speedup while maintaining generation quality, outperforming existing methods on LLaMA and Qwen models.
Leyline introduces a new serving-side primitive for managing KV cache in agentic LLMs, enabling efficient content editing and removal without full re-computation. The system uses declarative directives and RoPE-rotation corrections to handle policy-driven cache modifications, improving cache efficiency by 11.2 percentage points and agent solve rates by 14.3 percentage points.
