88 articles tagged with #llm-inference. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers analyzed data movement patterns in large-scale Mixture of Experts (MoE) language models (200B-1000B parameters) to optimize inference performance. Their findings led to architectural modifications achieving 6.6x speedups on wafer-scale GPUs and up to 1.25x improvements on existing systems through better expert placement algorithms.
Researchers developed Model2Kernel, a system that automatically detects memory safety bugs in CUDA kernels used for large language model inference. The system discovered 353 previously unknown bugs across popular platforms like vLLM and Hugging Face with only nine false positives.
Researchers introduce OnlineSpec, a framework that uses online learning to continuously improve draft models in speculative decoding for large language model inference acceleration. The approach leverages verification feedback to evolve draft models dynamically, achieving up to 24% speedup improvements across seven benchmarks and three foundation models.
Researchers conducted comprehensive benchmarks of LLM inference on AMD Instinct MI325X GPUs, testing models from 235B to 1 trillion parameters. The study reveals that architecture-aware optimization is critical, with different model types requiring specific configurations for optimal performance on AMD hardware.
Researchers have developed Zipage, a new high-concurrency inference engine for large language models that uses Compressed PagedAttention to solve memory bottlenecks. The system achieves 95% performance of full KV inference engines while delivering over 2.1x speedup on mathematical reasoning tasks.
Researchers developed a memory management system for multi-agent AI systems on edge devices that reduces memory requirements by 4x through 4-bit quantization and eliminates redundant computation by persisting KV caches to disk. The solution reduces time-to-first-token by up to 136x while maintaining minimal impact on model quality across three major language model architectures.
Researchers developed a multi-dimensional quality scoring framework for decentralized LLM inference networks that evaluates output quality across multiple dimensions including semantic quality and query-output alignment. The framework integrates with Proof of Quality (PoQ) mechanisms to provide better incentive alignment and defense against adversarial attacks in distributed AI compute networks.
Researchers propose an adaptive key-value caching strategy for large language models that dynamically allocates cache space based on recency and frequency patterns, improving upon traditional LRU eviction policies. The approach demonstrates up to 10.8% improvement in cache hit rates and 12.6% reduction in time-to-first-token on synthetic workloads, with more modest gains on real-world conversation data.
Researchers introduce UltraQuant, a 4-bit key-value cache compression technique optimized for long-context AI agents that need to process multiple conversation turns efficiently. The method achieves 3.47x faster response times in cache-pressured scenarios and 1.63x higher throughput compared to standard FP8 approaches, with practical optimizations for AMD GPU deployment.
Researchers introduce RKSC, a training-free inference framework that optimizes multi-step LLM reasoning by sharing KV cache across similar branches and implementing early exit mechanisms. The system achieves 3x average speedup over baseline methods with minimal error rates, advancing efficiency in large language model inference without requiring model retraining.
ReasonAlloc is a training-free framework that optimizes key-value cache memory allocation during LLM inference for reasoning tasks by using hierarchical, non-uniform budget distribution across layers and attention heads. The method significantly reduces memory bottlenecks in chain-of-thought reasoning while maintaining performance, outperforming existing compression approaches on mathematical reasoning benchmarks.
Researchers introduce AGENTSERVESIM, a hardware-aware simulator designed to evaluate serving policies for multi-turn LLM agents without requiring expensive accelerator deployments. The simulator accurately reproduces real-system performance within 6% error while running on standard CPUs, enabling scalable exploration of agent-serving policies across different hardware configurations and workload scenarios.
Harmonia is a new end-to-end RAG serving framework that optimizes the deployment and runtime performance of Retrieval-Augmented Generation pipelines. The system achieves 2.04x throughput improvements and reduces SLO violations by up to 78.4% through intelligent pipeline composition, heterogeneity-aware deployment, and dynamic load management.
Researchers propose Variational Speculative Decoding (VSD), a novel training method that improves LLM inference speed by optimizing draft models to better align with actual decoding requirements. By reformulating draft training as variational inference and incorporating path-level utilities, VSD achieves up to 9.6% speedup improvements over existing methods like EAGLE-3.
Researchers introduce Propagational Proxy Voting (PPV), an unsupervised aggregation method for multi-sample LLM inference that outperforms standard majority voting on MMLU-Pro benchmarks by leveraging semantic entropy and reasoning geometry signals. The method achieves +1.5 percentage point overall improvement and +2.24 pp on difficult questions without requiring labeled data or auxiliary training.
Researchers propose Semantic Cache Distillation (SCD), a technical framework that significantly reduces communication overhead in large language model inference by replacing raw Key-Value cache transmission with compact semantic codes. The method achieves up to 2.65x speedup in time-to-first-token while maintaining generation quality within 5% of baseline performance, addressing a critical bottleneck in disaggregated LLM serving architectures.
Researchers propose a fast matrix multiplication-based algorithm for matrix inversion in linear attention mechanisms, achieving up to 5x speedup on neural processing units while maintaining model accuracy under both standard and low-precision inference. The method addresses a critical computational bottleneck in long-context language modeling by using truncated Neumann expansion and parallel residual correction.
Researchers propose Budget-Guided MCTS, a tree-search algorithm that optimizes large language model inference by dynamically adjusting exploration and refinement strategies based on remaining token budgets. The method addresses a practical deployment challenge where fixed computational budgets vary across use cases, outperforming budget-agnostic approaches on mathematical and physics reasoning tasks.
Researchers demonstrate that exclusive batching (EB) can outperform the industry-standard mixed batching (MB) approach for LLM inference on bandwidth-constrained GPUs, with performance crossover dependent on hardware specifications and workload composition. A new hybrid scheduler (EB+) dynamically switches between strategies to optimize throughput across varying traffic conditions.
Researchers introduce Soft-NBCE, an improved method for processing ultra-long text contexts in large language models by replacing discrete chunk selection with weighted chunk fusion. The approach demonstrates measurable improvements on multi-hop reasoning tasks while maintaining efficient memory usage, addressing a critical bottleneck in LLM inference.
UniScale introduces a unified framework that combines model routing and test-time scaling to optimize large language model inference, balancing quality and computational cost. The system uses online learning via contextual multi-armed bandits to adapt inference policies dynamically, achieving fine-grained performance improvements over existing decoupled approaches.
A technical study reveals that batch-1 LLM inference on edge devices and robots is constrained by GPU launch overhead rather than memory bandwidth alone, with faster GPUs like the H100 achieving only 27% of theoretical peak bandwidth compared to 81% on slower L4 GPUs. Quantization techniques show inconsistent speedups, suggesting that hardware improvements don't automatically translate to latency gains without addressing software bottlenecks in physical AI deployments.
Researchers present methods for improving how large language models generate diverse pools of creative ideas during parallel inference without relying on seed examples. Their findings show that semantic direction stratification—organizing generations across different semantic directions with a single planning call—outperforms anchor-dependent baselines while maintaining quality and computational efficiency.
Researchers propose LaneRoPE, a novel technique that enables multiple parallel language model sequences to coordinate and share information during generation, improving reasoning accuracy without significant architectural changes or inference overhead.
EvoSpec introduces a dynamic framework for accelerating Large Language Model inference through real-time adaptation of vocabulary and parameters in speculative decoding. By addressing the vocabulary bottleneck that causes performance degradation in specialized domains, EvoSpec achieves 1.13x speedup improvements over static baselines while reducing memory overhead by 27%.
