109 articles tagged with #inference-efficiency. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
ConMem introduces a training-free framework for multi-agent systems that uses structured memory cards and relation-aware graphs to improve adaptation without additional training. The approach reduces inference overhead by over 80% and prunes more than 50% of candidate expansions while maintaining performance across multiple benchmarks.
Researchers introduce Projected Consistency Inference (PCI), a neural optimization method that solves the Traveling Salesman Problem more efficiently than gradient-based approaches by using structure-aware projections and local search instead of computationally expensive refinement. PCI achieves better optimality gaps (0.17% for 500 cities, 0.31% for 1000 cities) while reducing inference time by 30-40% compared to state-of-the-art FT2T methods.
Researchers introduce DyCP, a lightweight context management system that dynamically selects relevant dialogue segments for long-form conversations with large language models, improving inference efficiency without offline preprocessing. The method demonstrates competitive performance across multiple LLM benchmarks while reducing computational costs and latency in real-world dialogue applications.
Researchers introduce Ghosted Layers, a training-free method to recover performance degradation in layer-pruned large language models by solving an activation alignment problem through optimal linear operators. The technique uses a small calibration set to reconstruct hidden state mismatches introduced by pruning, maintaining efficiency gains while improving accuracy and perplexity across multiple LLM architectures.
Researchers introduce LEAP, a new technique for pruning large language models that uses learnable per-weight masks to achieve better accuracy than existing layer-wise methods, particularly at aggressive sparsity levels. The approach replaces earlier intractable parameterization methods with a Bernoulli-via-Gumbel-sigmoid relaxation, demonstrating 2.59 points average improvement over ADMM across multiple LLM families.
Researchers propose a consequence-aware compute allocation system for reasoning models that prioritizes high-impact tasks based on real-world failure costs rather than just predicted difficulty. Testing on software engineering benchmarks shows the method reduces cost-weighted loss by 22-33% compared to difficulty-based routing, with a practical predictor-driven variant retaining over 90% of theoretical gains.
Researchers introduce a differentiable Neural Architecture Search framework that jointly optimizes LLM architecture and mixed-precision quantization, achieving 1.4x faster inference speeds or 6% higher accuracy compared to sequential optimization approaches. This compression technique addresses the critical challenge of deploying large language models on edge devices without requiring extensive GPU training.
Researchers propose Upfront CoT (UCoT), a framework that compresses Chain-of-Thought reasoning in large language models by using a lightweight compressor to generate soft token representations of reasoning paths. The method maintains reasoning performance while reducing token usage by 50% on benchmarks, addressing the efficiency-performance tradeoff in advanced LLM inference.
Researchers propose EAPO, a reinforcement learning framework that teaches AI agents to use external tools selectively rather than excessively. The method improves accuracy while reducing redundant tool calls by 18-25% across multiple language models, demonstrating that agents can learn optimal tool-use patterns without compromising reasoning capabilities.
Researchers propose Base-Aligned Model Collaboration (BACo), an inference-time framework that dynamically combines base and aligned language models to improve both output diversity and quality simultaneously. The method uses token-level routing strategies based on uncertainty signals, achieving a 21.3% joint improvement in diversity-quality metrics without requiring expensive retraining or multi-pass decoding.
ConMoE presents a novel post-training compression method for Mixture-of-Experts language models that consolidates expert pools through prototype reassignment rather than pruning or weight merging. The train-free approach selectively retains pretrained experts as reusable prototypes and remaps original expert references to these prototypes, achieving competitive or superior performance on major MoE models while significantly reducing deployment memory requirements.
Researchers introduce OC-VTP, a lightweight vision token pruning method for Vision Language Models that reduces computational overhead by selectively retaining the most representative visual tokens without requiring model fine-tuning. The approach maintains inference accuracy across all pruning ratios while providing computational efficiency gains and interpretability benefits.
Researchers introduce ECHO, a novel test-time reinforcement learning algorithm that addresses rollout collapse and noisy pseudo-labels through entropy-confidence hybrid optimization. The method improves sampling efficiency and training robustness across mathematical and visual reasoning benchmarks while performing better under limited computational budgets.
Researchers present a new quantization method for large video diffusion models that achieves 59.3% memory reduction while maintaining near-baseline quality. The technique addresses challenges in compressing Wan2.2-I2V's mixture-of-experts architecture by using timestep-aware and expert-specific calibration strategies.
Researchers propose a critique-and-routing controller for multi-agent LLM systems that iteratively refines outputs through sequential decision-making rather than one-shot routing. The method uses reinforcement learning with agent-utilization constraints to achieve performance approaching the strongest agent while reducing computational calls by over 75%, advancing coordination efficiency in heterogeneous AI systems.
Researchers have released LLMSYS-HPOBench, the first comprehensive benchmark suite for hyperparameter optimization in real-world LLM systems, containing 364,450 configurations across 932 settings with multiple fidelity factors and cost metrics. The dataset addresses gaps in existing AutoML benchmarks by capturing the unprecedented complexity of optimizing both AI and non-AI components in production language model systems.
Researchers present a communication-theoretic framework that unifies LLM reliability techniques (retry, majority voting, self-consistency) under classical information theory, introducing a cost-aware router that achieves 56% lower costs than fixed approaches while maintaining quality. The work demonstrates that no single reliability technique dominates across all tasks, supporting dynamic per-task allocation strategies.
Researchers introduce COAST, a novel pruning framework for vision-language models that reduces visual tokens by 77.8% while maintaining 98.64% performance and achieving 2.15x speedup. Unlike existing methods that discard low-attention tokens, COAST uses adaptive semantic routing to preserve contextually essential information, preventing 'Visual Aphasia'—a failure mode where models lose visual grounding.
Researchers propose VecCISC, an optimization framework for weighted majority voting in large language models that reduces computational costs by 47% while maintaining accuracy. The method filters redundant or hallucinated reasoning traces using semantic similarity before evaluation, addressing the expensive overhead of confidence-scoring multiple candidate answers.
Researchers introduce Temporal Token Fusion (TTF), a training-free compression technique that reduces visual tokens in video-language models by 67% while maintaining 99.5% accuracy. The method addresses the critical bottleneck of LLM prefill costs in video understanding by identifying and fusing redundant tokens across video frames using local similarity matching.
Fluxion, a new hybrid CPU-GPU system, optimizes long-context inference by efficiently managing key-value caches split between host and GPU memory. The approach delivers 1.5x-3.7x speedup over existing baselines while maintaining near-baseline accuracy, addressing a critical bottleneck in modern large language model deployment.
Researchers introduce CA-SQL, an advanced Text-to-SQL pipeline that dynamically allocates computational resources based on task complexity to improve LLM reasoning. The method achieves state-of-the-art performance on the BIRD benchmark's challenging tier using only GPT-4o-mini, outperforming larger models and demonstrating the efficiency gains possible through intelligent inference-time optimization.
Researchers introduce MemSearcher, an AI agent framework that optimizes how large language models handle multi-turn interactions by maintaining compact memory instead of concatenating full conversation history. The approach uses a novel multi-context GRPO training method and demonstrates superior performance while maintaining stable token counts, reducing computational overhead.
BoostLoRA introduces a gradient-boosting framework that enables parameter-efficient fine-tuning adapters to grow their effective rank iteratively, allowing ultra-low-parameter models to match or exceed full fine-tuning performance across mathematical reasoning, code generation, and protein classification tasks. The method merges adapters with zero inference overhead while maintaining minimal per-round parameter costs.
Researchers demonstrate that reward-weighted classifier-free guidance (RCFG) can dynamically adjust autoregressive model outputs to optimize arbitrary reward functions at test time without retraining. Applied to molecular generation, this approach enables real-time optimization of competing objectives and accelerates reinforcement learning convergence when used as a teacher for policy distillation.
