297 articles tagged with #inference-optimization. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers propose Hybrid Verified Decoding, a technique that improves LLM inference speed by intelligently choosing between cache-based and model-based token drafting methods. The approach predicts draft acceptance rates before verification, achieving 2.73x average speedup on agentic workflows and outperforming existing methods like EAGLE3.
Researchers introduce Murmur, an inference system that optimizes long-form automatic speech recognition by balancing accuracy and latency through a two-level approach: intermediate chunk sizes at the inter-chunk level and attention sparsity exploitation at the intra-chunk level. The system achieves 4.2x latency reduction while maintaining single-pass accuracy on benchmark tests.
Researchers propose Chunk-Level Guided Generation, a training-free method using off-the-shelf large language models to score intermediate reasoning steps during small-model inference for mathematical problem-solving. The approach matches or outperforms specialized reward model-based systems on benchmarks like MATH and GSM8K without requiring expensive step-level training data.
Researchers introduce FLARE, a conversion framework that enables large language models with hybrid attention mechanisms to function as both autoregressive and diffusion models, addressing a key limitation in parallel decoding while maintaining model capability. The approach demonstrates competitive performance with existing diffusion language models while delivering throughput gains in concurrent serving scenarios.
Researchers introduce STaR-KV, a training-free compression framework that reduces key-value cache memory consumption in vision-language GUI agents by up to 40% while maintaining accuracy. The method addresses a critical bottleneck where models like UI-TARS-1.5-7B consume prohibitive GPU memory during multi-step interactions, enabling more practical deployment on standard accelerators.
Researchers introduce ProbScale, a framework that combines neural scaling laws with probing analysis to identify parameter-efficient subnetworks in Small Language Models. The method achieves 5-10x parameter reduction while maintaining 95-98% performance on downstream tasks, addressing deployment challenges for resource-constrained environments.
Researchers propose SimSD, a novel speculative decoding algorithm that enables diffusion language models to achieve up to 7.46x faster inference speeds while maintaining generation quality. By introducing a plug-and-play masking strategy, SimSD addresses the fundamental incompatibility between diffusion models' bidirectional attention and token-level speculative verification, a technique proven effective for autoregressive models.
Researchers propose a training-free weighted sampling framework using pretrained score-based generative models that achieves 1.2–4.7× speedups over existing methods. The approach avoids computationally expensive derivatives and resampling steps by incorporating lightweight guidance and adaptive scheduling, demonstrating effectiveness from synthetic experiments to large-scale applications like Stable Diffusion XL.
Researchers introduce T-POP, a novel algorithm that personalizes large language models in real-time by learning from user preference feedback during text generation, without requiring parameter updates or extensive pre-existing user data. The method combines test-time alignment with dueling bandits to efficiently balance exploration and exploitation, addressing the cold-start problem in LLM personalization.
Researchers introduce PETS, a framework for optimizing how many reasoning trajectories to sample from AI models during inference to maintain accuracy while reducing computational costs. By modeling trajectory allocation as a crowdsourcing problem, the approach achieves up to 75% budget savings on benchmarks while maintaining perfect consistency, addressing a key efficiency challenge in test-time scaling.
A controlled study examines how large-language-model agents perform with different skill documentation formats using SkillsBench, finding that skill availability dramatically improves task success (18-36 percentage points) while variations in presentation granularity produce minimal and uncertain effects across models.
Lumos-Nexus is a new video generation framework that separates training and inference to improve both reasoning quality and visual fidelity. The system uses a lightweight generator during training and progressively hands off to a high-capacity generator during inference through a technique called Unified Progressive Frequency Bridging, while introducing VR-Bench as a benchmark for reasoning-driven video generation.
Researchers propose Cross-Modal Attention Calibration (CMAC), a training-free method to reduce hallucinations in large vision-language models by addressing position bias and spurious correlations between visual and textual modalities. The approach combines an Inter-Modality Decoding module with contrastive mechanisms and a position calibration component to improve consistency between visual inputs and generated outputs.
Researchers propose Mixture of Horizons (MoH), a novel technique for vision-language-action models in robotics that processes action sequences at multiple time scales simultaneously to balance long-term planning with short-term precision. The method achieves state-of-the-art performance on robotic manipulation tasks, reaching 99% success rate on LIBERO benchmarks while enabling 2.5x faster inference through adaptive horizon selection.
Researchers introduce COVER, a new verification technique for diffusion language models that eliminates inefficient token oscillations during parallel decoding. By using KV cache overrides to preserve context while selectively verifying tokens in a single forward pass, COVER accelerates inference while maintaining output quality.
Researchers introduce Kinetic Path Energy (KPE), a physics-inspired metric for evaluating flow-based generative models that measures the dynamical effort of sampling trajectories. The analysis reveals a non-monotonic relationship between trajectory energy and generation quality, where excessive energy causes memorization rather than genuine generation, leading to a training-free inference method called Kinetic Trajectory Shaping that improves output fidelity.
Researchers propose block-based double decoders, a transformer architecture that combines the training efficiency of decoder-only models with the inference speed advantages of encoder-decoder models. The innovation uses doubly-causal block-based attention masks to enable full loss supervision and static sequence packing, achieving 2/3 reduction in KV-cache memory and per-token compute at inference time.
Researchers introduced ReasonOps, an unsupervised method for analyzing chain-of-thought traces from large language models that identifies seven universal reasoning operators (backtracking, inferring, hypothesizing, etc.) appearing consistently across 12 different LLM families. The framework enables model identification, correctness prediction, and early quality estimation without manual annotation, revealing that each model family has a distinctive reasoning fingerprint.
Researchers introduce CosmicFish-HRM, a compact language model that uses a Hierarchical Reasoning Module to dynamically adjust computational effort during inference based on input complexity. The approach challenges the assumption that larger models are necessary for advanced reasoning, suggesting adaptive computation depth could offer efficiency gains as model scale increases.
BlockBatch introduces a training-free inference framework that optimizes diffusion language models by executing multiple block-size branches simultaneously, achieving 26.6% reduction in computational steps and 1.33x speedup over existing methods. The approach exploits the complementary nature of different decoding granularities to balance parallelism with accuracy while managing the inherent trade-offs in block-wise inference.
Researchers introduce SWAI, a training-free method for controlling language model outputs by manipulating logit scores using corpus-derived statistics. The technique enables real-time steering of model behavior—such as adjusting readability, politeness, and toxicity—without modifying model weights or accessing internal layers, outperforming existing prompt-based and logit-level baselines.
Chinese researchers have developed an AI model that leverages idle processing time to predict and prepare for users' next queries before they're asked. This advancement in predictive AI could reduce latency and improve user experience by pre-computing likely requests during periods when the system would otherwise be inactive.
Researchers analyzed backtracking patterns in reasoning traces from the Qwen3-8B model, finding that correct reasoning typically shows early, isolated self-corrections while incorrect reasoning exhibits persistent, clustered revisions occurring late in traces. The study demonstrates that burst-aware filtering of reasoning traces can improve model reliability by identifying unstable reasoning patterns before completion.
Researchers introduced HRBench, a unified evaluation framework for testing hybrid-reasoning LLMs that allow dynamic switching between fast and slow reasoning modes. The framework systematically compares 12+ prior methods across three switching strategy families and four training approaches, revealing that prompt-based methods offer better token-accuracy trade-offs while routing methods provide more stable cost reduction.
Researchers introduce DREAM-R, a framework that accelerates reasoning in multimodal AI models through improved speculative execution. The system uses reinforcement learning to align draft models with target reasoning, a verification mechanism to prevent errors, and parallel processing to achieve significant speedup while maintaining accuracy.
