171 articles tagged with #inference-optimization. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers introduce Mixed-Policy Distillation (MPD), a technique that compresses reasoning in smaller language models by having larger teacher models rewrite student-generated reasoning traces into more concise versions. The method reduces token usage by up to 27.1% while maintaining or improving performance, addressing critical deployment constraints around memory, latency, and serving costs.
SynerDiff is a new continuous batching system for diffusion model inference that addresses resource contention issues between UNet and VAE components. The system achieves 1.6× throughput improvement and up to 78.7% latency reduction through intra-level and inter-level optimization strategies, enabling faster AI-generated content services.
LoopVLA introduces a recurrent Vision-Language-Action model architecture that learns when to stop refining representations for robotic control tasks, achieving 45% parameter reduction and 1.7x faster inference while maintaining or improving task performance. The model uses self-supervised learning to estimate representation sufficiency rather than relying on predefined layer depths or heuristic rules.
Researchers introduce SciAidanBench, a benchmark revealing that LLM capability improvements are uneven across tasks and domains—a phenomenon termed 'jaggedness.' By evaluating 19 models across 8 providers, they demonstrate that stronger models don't uniformly excel at scientific creativity, but this fragmentation can be leveraged through ensemble methods to achieve superior performance.
Researchers introduce Priming, a method that converts pre-trained Transformers into efficient Hybrid State-Space models through knowledge transfer rather than training from scratch. The technique recovers downstream performance using less than 0.5% of original pre-training tokens and enables the first large-scale comparison of SSM architectures, with Hybrid GKA 32B achieving 3.8-point reasoning improvements while delivering 2.3x faster decoding.
Researchers introduce KeyStone, an inference-time method that improves physical AI model performance by generating multiple candidate action trajectories in parallel and selecting the most physically coherent one using geometric clustering. The technique achieves up to 13.3% improvement in task success rates across vision-language-action and world-action models without additional latency or training costs.
Researchers introduce SpikingBrain, a family of brain-inspired large language models optimized for efficient long-context processing on non-NVIDIA hardware. The models achieve comparable performance to Transformers while requiring significantly fewer tokens for training, delivering up to 100x speedup for long sequences and 69% sparsity for low-power operation.
Researchers introduce Cached State Representation (CSR), a framework that reduces latency in deploying large language models for robotics by 26-fold through optimized token caching and asynchronous state management. The approach enables real-time robot control with massive language models while maintaining full contextual understanding over infinite operational horizons.
Researchers introduce CASCADE, a framework enabling large language models to continuously learn and improve during deployment without modifying parameters, using an episodic memory system formulated as a contextual bandit problem. The approach demonstrates 20.9% improvement over zero-shot prompting across 16 diverse tasks, addressing a fundamental limitation in current LLM lifecycles where learning stops after training ends.
Researchers introduce GASim, a graph-accelerated framework that combines large language models with agent-based models for large-scale social simulations. The system achieves 9.94x speedup and reduces computational token usage by 80% while maintaining accuracy in modeling real-world opinion dynamics.
Researchers introduce TAPER, an admission controller for managing parallel branch execution in LLM serving systems. The system dynamically regulates how many concurrent decoding branches are allowed per request step, balancing throughput gains against degradation to co-batched requests, achieving 1.77x improvement in goodput over conservative baselines.
Researchers propose SARQC, a new post-training quantization framework for large language models that adds saliency-aware regularization to prevent quantized weights from drifting too far from original values. The method improves generalization performance across dense and mixture-of-experts LLMs without increasing inference costs.
Researchers introduce Post-Reasoning, a technique that improves LLM performance by having models justify answers after generating final responses, without increasing latency or token costs. The method demonstrates 17.37% mean performance improvements across 117 model-benchmark settings and establishes a new efficiency frontier for direct-answer AI capabilities.
Researchers propose constant-context skill learning, a framework enabling LLM agents to learn reusable task procedures as lightweight modules rather than storing long prompts in memory. The approach reduces token usage per inference by 2-7x while maintaining or improving performance across multiple benchmark environments, addressing the privacy-capability tradeoff in agent deployment.
Researchers introduce SPEED, a novel inference optimization technique for long-context language models that reduces computational cost by materializing key-value cache states only in lower layers during the prefill phase while maintaining full-depth processing during decoding. Testing on Llama-3.1-8B demonstrates 33% improvement in time-to-first-token, 22% improvement in tokens-per-second, and 25% reduction in KV memory with minimal quality degradation, suggesting that prompt tokens don't require persistent full-depth caching.
ReFlect introduces a training-free harness system that wraps around LLMs to detect and recover from reasoning failures in complex, multi-step tasks. Testing across six models shows significant improvements in task success rates, with gains inversely correlated to baseline performance, though the approach reveals limitations in how smaller models handle structured reasoning.
Researchers introduce FASQ, a calibration-free compression framework for large language models that uses product quantization to achieve flexible compression ratios between 27-49% of original model size. The method outperforms existing quantization approaches like GPTQ and AWQ while enabling faster inference than FP16 on consumer GPUs through custom CUDA kernels.
Researchers introduced Uno-Orchestra, a new orchestration framework for multi-agent LLM systems that dynamically decides when to decompose tasks and which model-primitive pairs to use, achieving 77% accuracy across 13 benchmarks while reducing computational costs by an order of magnitude compared to existing approaches.
TokenArena introduces a continuous benchmark framework that evaluates AI inference endpoints across energy efficiency, latency, cost, and output quality rather than just model-level comparisons. Testing 78 endpoints across 12 model families reveals dramatic performance variance—the same model differs by up to 12.5 accuracy points and 6.2x in energy efficiency depending on deployment configuration, with workload type fundamentally reordering cost-effectiveness rankings.
Researchers introduce BWLA, a post-training quantization framework that achieves 1-bit weight compression alongside low-bit activations for large language models, addressing a critical bottleneck in LLM deployment. The method delivers 3.26× inference speedup on Qwen3-32B while maintaining competitive accuracy, potentially enabling more efficient LLM inference across resource-constrained environments.
Researchers introduce Disentangled Safety Adapters (DSA), a modular framework that decouples safety mechanisms from base AI models using lightweight adapters. The approach achieves superior safety performance with minimal inference overhead while enabling dynamic, context-dependent alignment adjustments at inference time.
Researchers propose LightKV, a technique that reduces Key-Value cache memory overhead in Large Vision-Language Models by compressing vision tokens using cross-modality message passing guided by text prompts. The method achieves 50% reduction in KV cache size while using only 55% of original vision tokens and reducing computation by up to 40%, maintaining performance across eight benchmark datasets.
Researchers introduce Efficient-DLM, a framework for converting pretrained autoregressive language models into diffusion language models that enable parallel, non-autoregressive generation. The approach uses block-wise attention patterns and position-dependent masking to preserve model accuracy while achieving 4.5x higher throughput compared to existing models.
Researchers propose Path-Lock Expert (PLE), an architectural solution that separates reasoning and non-reasoning modes in hybrid-thinking language models by replacing single MLPs with two specialized experts. The approach significantly reduces reasoning leakage in non-reasoning mode while maintaining strong performance in reasoning tasks, suggesting that controllable hybrid thinking is fundamentally an architectural problem rather than a training problem.
Researchers introduce Vec-LUT, a novel vector-based lookup table technique that dramatically improves ultra-low-bit LLM inference on edge devices by addressing memory bandwidth underutilization. The method achieves up to 4.2x performance improvements over existing approaches, enabling faster LLM execution on CPUs than specialized NPUs.
