64 articles tagged with #gpu-optimization. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers introduce StreamKL, a novel GPU optimization for computing KL divergence in attention distillation that reduces memory requirements from O(N_Q N_K) to O(1) and delivers up to 43x forward-pass speedups. This advancement enables efficient knowledge distillation and model compression for long-context language models on standard hardware.
Researchers introduce ICALens, a new method for interpreting language model representations using independent component analysis (ICA) instead of expensive sparse autoencoders (SAEs). The approach efficiently recovers interpretable directions without requiring large neural dictionary training, achieving competitive performance on standard benchmarks while offering a faster, more accessible alternative for LLM analysis.
Researchers have developed a method to improve multi-GPU machine learning training by enabling computation and communication to execute simultaneously using shared-memory allocation and scheduling priority adjustments. The technique demonstrates up to 25.5% execution time reduction across NVIDIA and AMD GPUs without requiring modifications to vendor libraries.
Researchers introduce STAR-KV, an adaptive compression framework that reduces KV cache memory requirements in large language models by up to 75% through low-rank projections and intelligent rank selection. The technique achieves up to 20x compression when combined with quantization and delivers significant speedups in attention computation, addressing a critical bottleneck in LLM inference efficiency.
Researchers introduce APEX4, a pure INT4 inference system that addresses the long-standing challenge of W4A4 quantization in large language models by adapting compute strategies based on GPU architecture. The system achieves up to 2.09× speedup on consumer GPUs while maintaining quality within 0.63 perplexity points of FP16 baselines, making efficient LLM inference more practical across diverse hardware platforms.
Researchers introduce E2Former-V2, a more scalable architecture for Equivariant Graph Neural Networks that models 3D molecular systems. By combining algebraic sparsity with hardware-optimized execution, the model achieves 20× computational improvements while maintaining competitive accuracy on molecular datasets.
Researchers introduce HASTE, a hardware-aware sparse training method for extreme multi-label classification that uses group-shared fixed fan-in sparsity to optimize GPU execution. The approach achieves up to 25x speedup in backward passes compared to standard sparse methods while maintaining competitive accuracy, addressing the memory-compute bottleneck in models with millions of output labels.
Researchers introduce APB-V, a sequence-parallel framework that accelerates long-video inference in Large Multimodal Models by distributing approximate attention across multiple GPUs. The approach achieves 12.72x speedup over FlashAttn while processing longer videos without visual compression, addressing a critical bottleneck in AI video understanding.
Researchers present an efficient vision-language model for generating pathology reports from whole-slide images (WSIs), achieving 64x sequence length reduction through optimized patch sampling while requiring only half an NVIDIA H100 GPU for training. The two-stage approach combines WSI captioning with case-level fine-tuning to handle multi-slide pathology cases, establishing a reproducible baseline for resource-constrained medical AI development.
Researchers introduce Parallel Variational Monte Carlo (PVMC), a novel training method for deep state space models that combines strengths of variational and sequential Monte Carlo approaches. The technique achieves comparable or superior performance to existing methods while running 10x faster, addressing a critical scalability bottleneck in training complex temporal models.
Researchers develop GPU kernel optimizations for Graph Neural Networks that reduce memory traffic and improve computational efficiency across three major layer types. The work achieves significant speedups (up to 8.5x for GATv2, 10x for aggregation layers) while dramatically reducing memory consumption, with implementations released as drop-in replacements for existing frameworks.
SANA-Streaming introduces a real-time video editing system that achieves 24 FPS at 1280x704 resolution on consumer GPUs through a hybrid diffusion transformer architecture and specialized optimization for NVIDIA hardware. The breakthrough combines algorithmic improvements in temporal consistency with system-level co-design, enabling practical applications in live broadcasting and gaming that were previously computationally infeasible.
Researchers demonstrate that large language models can effectively forecast GPU kernel performance, reducing expensive on-device evaluations during optimization searches. By acting as selective surrogates that know their confidence limits, LLMs enable kernel searches to evaluate multiple candidates under fixed GPU budgets, ultimately discovering faster kernels than baseline approaches.
PromptEmbedder introduces a dual-LLM framework that decouples text embedding from specific model architectures, achieving comparable performance to LoRA while reducing GPU memory by 40% and accelerating training 3.7x. The innovation enables efficient transfer across different LLM backbones by retraining only a lightweight alignment matrix rather than entire models.
Researchers present a systematic study of Attention-FFN Disaggregation (AFD), a technique that separates attention and expert layers across different GPU groups to optimize inference serving for Mixture-of-Experts language models. The framework demonstrates that AFD enables 4k tokens/s throughput on DeepSeek-V3.2 under strict latency constraints where traditional disaggregation approaches fail, providing design principles for scaling LLM infrastructure.
Researchers introduce Qrita, an efficient algorithm for Top-k and Top-p sampling in large language models that uses pivot-based truncation instead of sorting. The method achieves 1.4x throughput improvements with 50% less memory usage while maintaining identical output to traditional sorting approaches, and has been adopted as the default sampler in vLLM.
FlashSVD v1.5 addresses a critical gap between theoretical and practical performance gains in SVD-compressed transformer inference, delivering up to 2.55x speedup through runtime optimization rather than algorithmic improvements alone. The work demonstrates that low-rank compression benefits require co-designed inference systems to translate parameter reduction into actual serving speed improvements.
Researchers introduce MISA, an optimization technique that reduces computational costs in DeepSeek's sparse attention mechanism for large language models by treating indexer heads as a mixture-of-experts system. The method achieves 3.82x speedup on GPU inference while maintaining performance across benchmarks, addressing a key bottleneck in long-context LLM processing.
Researchers propose ESSAM, a novel training framework combining Evolution Strategies with Sharpness-Aware Maximization to fine-tune large language models for mathematical reasoning while dramatically reducing GPU memory requirements. The approach achieves comparable accuracy to reinforcement learning methods like PPO and GRPO while using 18-10× less memory, addressing a critical bottleneck in LLM development.
Researchers introduce a queueing-theoretic framework that models LLM inference stability by accounting for both computational and GPU memory constraints from KV caching. The framework derives conditions for service stability and enables operators to calculate optimal cluster sizes for efficient GPU provisioning, with experimental validation showing predictions within 10% accuracy.
Researchers introduce RoundPipe, a novel pipeline scheduling algorithm that enables efficient fine-tuning of large language models on consumer-grade GPUs by eliminating the weight binding constraint that causes computational bottlenecks. The system achieves 1.48-2.16x speedups over existing approaches and enables fine-tuning of models with up to 235 billion parameters on standard hardware.
Researchers identify dimensional misalignment as a critical bottleneck in compressed large language models, where parameter reduction fails to improve GPU performance due to hardware-incompatible tensor dimensions. They propose GAC (GPU-Aligned Compression), a new optimization method that achieves up to 1.5× speedup while maintaining model quality by ensuring hardware-friendly dimensions.
Researchers introduce Deep Optimizer States, a technique that reduces GPU memory constraints during large language model training by dynamically offloading optimizer state between host and GPU memory during computation cycles. The method achieves 2.5× faster iterations compared to existing approaches by better managing the memory fluctuations inherent in transformer training pipelines.
TensorHub introduces Reference-Oriented Storage (ROS), a novel weight transfer system that enables efficient reinforcement learning training across distributed GPU clusters without physically copying model weights. The production-deployed system achieves significant performance improvements, reducing GPU stall time by up to 6.7x for rollout operations and improving cross-datacenter transfers by 19x.
Researchers have developed a new low-bit mixed-precision attention kernel called Diagonal-Tiled Mixed-Precision Attention (DMA) that significantly speeds up large language model inference on NVIDIA B200 GPUs while maintaining generation quality. The technique uses microscaling floating-point (MXFP) data format and kernel fusion to address the high computational costs of transformer-based models.
