62 articles tagged with #inference-efficiency. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers introduce OccamToken, a training-free method for compressing vision-language models by pruning unnecessary visual tokens while maintaining accuracy. The approach reduces visual token sequences by 98.6% (from 2,880 to 40 tokens) on LLaVA-NeXT while preserving over 93% accuracy, addressing computational bottlenecks in VLM inference.
Researchers propose modeling Large Reasoning Models' Chain-of-Thought processes as trajectories through a six-state Finite State Machine, enabling better understanding and control of reasoning dynamics. They introduce Q-Value guided steering, a training-free method that optimizes reasoning by applying sparse activation steering at sentence boundaries, achieving significant performance gains across multiple benchmarks with minimal computational overhead.
ParaTool is a new framework that shifts tool representations from context to parameters in large language models, enabling efficient tool calling without relying on lengthy in-context documentation. The approach uses parametric tool pre-training, soft tool selection, and fine-tuning to reduce inference overhead and hallucination risks while maintaining superior performance on benchmark tests.
Researchers introduce Logit-aware Final-block Quantization (LFQ), a technique that improves low-bit quantization of large language models by optimizing the final transformer block to preserve token probability distributions. This advancement addresses quality degradation in generative tasks while maintaining efficiency gains critical for deploying scaled LLMs.
Researchers introduce CFGzip, a token space compression technique that dramatically accelerates constrained decoding for large language models using context-free grammars. The method achieves up to 100x latency reduction and 7.5x total speedup, making complex grammar-constrained generation feasible at scale.
Researchers identify a critical failure mode in large reasoning models where they detect insufficient information but still produce unsupported answers instead of abstaining. The proposed Judge-Then-Solve (JTS) framework trains models to make explicit answerability commitments before reasoning, significantly improving safe abstention rates and inference efficiency.
PrunePath is a new structured sparsification framework that optimizes feed-forward networks in language models by replacing traditional pruning methods with a softmax-normalized routing system. The approach converts model sparsity into practical hardware efficiency gains, demonstrated through memory savings and faster decoding speeds via custom Triton kernels.
Researchers develop a systematic approach to quantization-aware training for large language models using 8-bit floating-point formats, identifying and solving two critical failure modes—amax saturation and catastrophic forgetting—that don't surface in standard training metrics. Their solution achieves near-lossless performance with only 0.43% degradation on benchmark tasks, advancing practical LLM deployment efficiency.
Researchers conducted an extensive empirical study evaluating FP8, INT8, and INT4 quantization formats across the Llama-3.1 model family, finding that FP8 is effectively lossless while INT4 weight-only quantization performs surprisingly well. The findings provide practical deployment guidelines for optimizing the accuracy-performance trade-off in large language model inference at scale.
Researchers propose STARS, a training framework that stabilizes Looped Language Models (LoopLMs) to enable reliable test-time scaling through latent reasoning. The method uses Jacobian Spectral Radius Regularization to constrain neural states toward stable fixed points, addressing a critical problem where model performance peaks then collapses with increased recurrence depth.
Echo-LoRA introduces a parameter-efficient fine-tuning method that injects cross-layer representations from deeper neural network layers into shallow LoRA modules during training, achieving 3-5.7% performance improvements on reasoning tasks without adding inference costs. The technique discards its auxiliary training path post-deployment, maintaining the efficiency benefits of standard LoRA while delivering measurable capability gains.
Researchers propose RDKV, a novel compression technique that jointly optimizes eviction and quantization of the Key-Value cache in large language models to reduce memory bottlenecks during inference. The method achieves 4.5x decode speedup and 1.9x peak memory reduction on 128K context lengths while maintaining 97.81% accuracy, addressing a critical performance constraint in LLM deployment.
Researchers demonstrate that Mixture of Experts (MoE) models contain substantial underutilized sparsity within individual experts that can be exploited without modifying model parameters. By implementing intra-expert activation sparsity in vLLM, they achieve up to 2.5x speedup in MoE layer execution, offering a practical optimization path for efficient large language model deployment.
Researchers propose LEAD, a new method that makes large reasoning AI models more efficient by dynamically balancing accuracy and output length during training. Unlike existing approaches using static constraints, LEAD adapts per-problem length targets and reward calibration in real-time, achieving better accuracy and shorter outputs across mathematical reasoning benchmarks.
Researchers propose Kaczmarz Linear Attention (KLA), an improved algorithm for long-context language modeling that replaces empirically-learned coefficients with mathematically-derived key-norm-normalized step sizes. KLA outperforms existing linear attention baselines like Gated DeltaNet while maintaining computational efficiency and enabling stable processing of up to 65K token contexts.
Researchers introduce RuPLaR, a novel compression framework that enables Large Language Models to generate latent reasoning tokens in a single training stage, eliminating inefficiencies of traditional multi-step Chain-of-Thought approaches. The method achieves 11.1% accuracy improvement over existing latent CoT systems while using minimal tokens, demonstrating significant progress in efficient LLM reasoning.
Researchers introduce Adaptive Reparameterized Time (ART), a reinforcement learning approach that optimizes timestep scheduling for diffusion models to improve sample generation efficiency. The method reduces computational costs while maintaining image quality, with demonstrated improvements on benchmark datasets and cross-dataset transferability.
Switchcraft is a new AI model router specifically designed for agentic tool calling that selects the lowest-cost model while maintaining correctness. The system achieves 82.9% accuracy matching top models while reducing inference costs by 84%, demonstrating that larger models don't consistently outperform smaller ones on function-calling tasks.
Researchers introduce CASPO, a framework that improves reasoning reliability in large language models by aligning token-level confidence with step-wise logical correctness through preference optimization. The method achieves better performance than tree-search approaches without requiring separate reward models, while introducing CaT inference that dynamically prunes uncertain reasoning branches with minimal computational overhead.
Researchers introduce LaProx, a novel KV Cache eviction strategy for long-context LLM inference that reformulates the problem from head-wise weight averaging to output-aware layer-wise matrix multiplication. The method achieves 2× accuracy loss reduction under extreme compression while maintaining performance with just 5% of the original KV cache.
Researchers present Trajectory-Shaped Discrete Flow Matching (TS-DFM), a technique that improves text generation efficiency by using an energy-based guidance system during training to select better token transformation paths. The method enables a compact student model to achieve 32% lower perplexity than a 1,024-step teacher while running 128x faster at just 8 steps, setting new benchmarks for discrete generation tasks.
Researchers introduce Recursive Agent Optimization (RAO), a reinforcement learning method enabling AI agents to spawn and delegate tasks to themselves recursively. This approach allows agents to handle longer contexts, solve harder problems through divide-and-conquer strategies, and achieve better training efficiency with reduced computational time.
TSCG is a deterministic compiler that converts JSON tool schemas into structured text optimized for language model interpretation, solving a critical failure point in agentic AI systems. The technology restores accuracy in smaller models (4B-14B) from near-zero to 84%+ on production-scale tool catalogs while reducing token consumption by 52-57%, shipping as a lightweight TypeScript package.
RetentiveKV introduces an entropy-driven optimization method for multimodal large language models that achieves 5x KV cache compression and 1.5x decoding acceleration by reformulating token eviction as continuous memory evolution rather than discrete pruning. The approach addresses limitations of existing compression methods by accounting for visual tokens that gain importance later in decoding and preserving spatial continuity of visual information.
Researchers present a decision-making framework to optimize when large language models should call external tools like web search. The study reveals that models often misjudge their actual need for tool use, and proposes lightweight estimators trained on hidden states to improve tool-calling decisions, demonstrating performance gains across multiple tasks.
