AIBullisharXiv – CS AI · Jun 237/10
🧠Researchers propose Geometry-Aware Online Scheduling, introducing the Smallest Volume First (SVF) algorithm to optimize LLM inference by accounting for dynamic memory footprint of Key-Value caches. The approach improves upon traditional time-centric scheduling heuristics, achieving significant reductions in latency and throughput gains when integrated into vLLM.
🧠 Llama
AIBullisharXiv – CS AI · Jun 97/10
🧠Researchers introduce APEX4, a pure INT4 inference system that addresses the long-standing challenge of W4A4 quantization in large language models by adapting compute strategies based on GPU architecture. The system achieves up to 2.09× speedup on consumer GPUs while maintaining quality within 0.63 perplexity points of FP16 baselines, making efficient LLM inference more practical across diverse hardware platforms.
$ADA🏢 Perplexity
AIBullisharXiv – CS AI · Jun 27/10
🧠Researchers introduce Lodestar, a machine learning-based request routing system that dynamically assigns large language model inference tasks to GPU instances in distributed clusters. The system achieves up to 4.38x improvements in latency metrics compared to existing heuristics by continuously learning optimal routing strategies in real-time.
AIBullisharXiv – CS AI · May 287/10
🧠GroundedCache proposes a safety-first framework for reusing cached answers in retrieval-augmented generation systems by validating four conditions before serving cached responses. The system achieves near-zero unsafe-served rates (0-1.5%) across benchmarks while maintaining minimal latency overhead, addressing critical vulnerabilities in current caching approaches that can serve incorrect answers.
AIBullisharXiv – CS AI · May 277/10
🧠Researchers introduce Qrita, an efficient algorithm for Top-k and Top-p sampling in large language models that uses pivot-based truncation instead of sorting. The method achieves 1.4x throughput improvements with 50% less memory usage while maintaining identical output to traditional sorting approaches, and has been adopted as the default sampler in vLLM.
AIBullisharXiv – CS AI · May 127/10
🧠Researchers demonstrate that Mixture of Experts (MoE) models contain substantial underutilized sparsity within individual experts that can be exploited without modifying model parameters. By implementing intra-expert activation sparsity in vLLM, they achieve up to 2.5x speedup in MoE layer execution, offering a practical optimization path for efficient large language model deployment.
AIBullisharXiv – CS AI · Apr 207/10
🧠Researchers introduced Ragged Paged Attention (RPA), a specialized inference kernel optimized for Google's TPUs that enables efficient large language model deployment. The innovation addresses the GPU-centric design of existing LLM serving systems by implementing fine-grained tiling and custom software pipelines, achieving up to 86% memory bandwidth utilization on TPU hardware.
🧠 Llama
AIBullisharXiv – CS AI · Apr 147/10
🧠Researchers introduce SPEED-Bench, a comprehensive benchmark suite for evaluating Speculative Decoding (SD) techniques that accelerate LLM inference. The benchmark addresses critical gaps in existing evaluation methods by offering diverse semantic domains, throughput-oriented testing across multiple concurrency levels, and integration with production systems like vLLM and TensorRT-LLM, enabling more accurate real-world performance measurement.
AIBullisharXiv – CS AI · Mar 277/10
🧠Researchers developed Model2Kernel, a system that automatically detects memory safety bugs in CUDA kernels used for large language model inference. The system discovered 353 previously unknown bugs across popular platforms like vLLM and Hugging Face with only nine false positives.
🏢 Hugging Face
AIBullisharXiv – CS AI · Mar 177/10
🧠Justitia is a new scheduling system for task-parallel LLM agents that optimizes GPU server performance through selective resource allocation based on completion order prediction. The system uses memory-centric cost quantification and virtual-time fair queuing to achieve both efficiency and fairness in LLM serving environments.
🏢 Meta
AINeutralarXiv – CS AI · Mar 127/10
🧠Researchers conducted comprehensive benchmarks of LLM inference on AMD Instinct MI325X GPUs, testing models from 235B to 1 trillion parameters. The study reveals that architecture-aware optimization is critical, with different model types requiring specific configurations for optimal performance on AMD hardware.
🧠 Llama
AIBullisharXiv – CS AI · Jun 236/10
🧠Researchers propose an adaptive key-value caching strategy for large language models that dynamically allocates cache space based on recency and frequency patterns, improving upon traditional LRU eviction policies. The approach demonstrates up to 10.8% improvement in cache hit rates and 12.6% reduction in time-to-first-token on synthetic workloads, with more modest gains on real-world conversation data.
AIBullisharXiv – CS AI · Jun 196/10
🧠Researchers introduce UltraQuant, a 4-bit key-value cache compression technique optimized for long-context AI agents that need to process multiple conversation turns efficiently. The method achieves 3.47x faster response times in cache-pressured scenarios and 1.63x higher throughput compared to standard FP8 approaches, with practical optimizations for AMD GPU deployment.
AIBullisharXiv – CS AI · May 96/10
🧠VibeServe introduces an AI-driven approach to LLM serving infrastructure that automatically generates specialized system stacks for different workloads rather than relying on single general-purpose designs. The system matches vLLM performance in standard deployment scenarios while significantly outperforming existing solutions in non-standard cases, suggesting a paradigm shift toward generation-time specialization in infrastructure software.
AIBullisharXiv – CS AI · Mar 36/104
🧠Researchers have developed EasySteer, a unified framework for controlling large language model behavior at inference time that achieves 10.8-22.3x speedup over existing frameworks. The system offers modular architecture with pre-computed steering vectors for eight application domains and transforms steering from a research technique into production-ready capability.
AIBullishHugging Face Blog · Jun 36/105
🧠The article discusses optimizing GPU efficiency using co-located vLLM (virtual Large Language Model) infrastructure in TRL (Transformer Reinforcement Learning). This approach aims to maximize GPU utilization and reduce computational waste in AI model training and deployment.
AIBullishHugging Face Blog · Jan 166/106
🧠Text Generation Inference introduces multi-backend support for TRT-LLM and vLLM, expanding deployment options for AI text generation models. This development enhances flexibility and performance optimization capabilities for developers working with large language models.
AINeutralHugging Face Blog · Oct 31/106
🧠The article title suggests a discussion about Very Large Language Models (VLLMs) and evaluation methodologies, but the article body appears to be empty or not provided.
