273 articles tagged with #optimization. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers introduce MASPOB, a bandit-based framework that optimizes prompts for Multi-Agent Systems using Graph Neural Networks to handle topology-induced coupling. The system reduces search complexity from exponential to linear while achieving state-of-the-art performance across benchmarks.
Research reveals an exponential gap between structured and unstructured neural network pruning methods. While unstructured weight pruning can approximate target functions with O(d log(1/ε)) neurons, structured neuron pruning requires Ω(d/ε) neurons, demonstrating fundamental limitations of structured approaches.
Researchers propose MIStar, a memory-enhanced improvement search framework using heterogeneous graph neural networks for flexible job-shop scheduling problems in smart manufacturing. The approach significantly outperforms traditional heuristics and state-of-the-art deep reinforcement learning methods in optimizing production schedules.
Researchers propose a new quantum annealing framework for training CNN classifiers that avoids gradient-based optimization by using Quadratic Unconstrained Binary Optimization (QUBO). The method shows competitive performance with classical approaches on image classification benchmarks while remaining compatible with current D-Wave quantum hardware.
Researchers introduce Neural Paging, a new architecture that addresses the computational bottleneck of finite context windows in Large Language Models by implementing a hierarchical system that decouples reasoning from memory management. The approach reduces computational complexity from O(N²) to O(N·K²) for long-horizon reasoning tasks, potentially enabling more efficient AI agents.
Researchers identified a structural misalignment in Transformer models where residual connections tie to current tokens while supervision targets next tokens. They propose lightweight residual attenuation techniques that improve autoregressive Transformer performance by addressing this input-output alignment shift.
CSRv2 introduces a new training approach for ultra-sparse embeddings that reduces inactive neurons from 80% to 20% while delivering 14% accuracy gains. The method achieves 7x speedup over existing approaches and up to 300x improvements in compute and memory efficiency compared to dense embeddings.
Researchers introduce DRAGON, a new framework that combines Large Language Models with metaheuristic optimization to solve large-scale combinatorial optimization problems. The system decomposes complex problems into manageable subproblems and achieves near-optimal results on datasets with over 3 million variables, overcoming the scalability limitations of existing LLM-based solvers.
Researchers introduce AgentOCR, a framework that converts AI agent interaction histories from text to compressed visual format, reducing token usage by over 50% while maintaining 95% performance. The system uses visual caching and adaptive compression to address memory bottlenecks in large language model deployments.
Researchers introduce ExGRPO, a new framework that improves AI reasoning by reusing and prioritizing valuable training experiences based on correctness and entropy. The method shows consistent performance gains of +3.5-7.6 points over standard approaches across multiple model sizes while providing more stable training.
Researchers introduce the first theoretical framework analyzing convergence of adaptive optimizers like Adam and Muon under floating-point quantization in low-precision training. The study shows these algorithms maintain near full-precision performance when mantissa length scales logarithmically with iterations, with Muon proving more robust than Adam to quantization errors.
Researchers prove that gradient descent in neural networks converges to optimal robustness margins at an extremely slow rate of Θ(1/ln(t)), even in simplified two-neuron settings. This establishes the first explicit lower bound on convergence rates for robustness margins in non-linear models, revealing fundamental limitations in neural network training efficiency.
Researchers introduce HEAPr, a novel pruning algorithm for Mixture-of-Experts (MoE) language models that decomposes experts into atomic components for more precise pruning. The method achieves nearly lossless compression at 20-25% pruning ratios while reducing computational costs by approximately 20%.
Researchers introduce REMS, a unified framework for solving combinatorial optimization problems that views problems as resource allocation tasks. The framework enables reusable metaheuristic algorithms and outperforms established solvers like GUROBI and SCIP on large-scale instances across 10 different problem types.
Researchers demonstrate that training loss curves for large language models can collapse onto universal trajectories when hyperparameters are optimally set, enabling more efficient LLM training. They introduce Celerity, a competitive LLM family developed using these insights, and show that deviation from collapse can serve as an early diagnostic for training issues.
Researchers developed ZeroDVFS, a system that uses Large Language Models to optimize power management in embedded systems without requiring extensive profiling. The system achieves 7.09 times better energy efficiency and enables zero-shot deployment for new workloads in under 5 seconds through LLM-based code analysis.
Researchers have developed Curvature-Aware Policy Optimization (CAPO), a new algorithm that improves training stability and sample efficiency for Large Language Models by up to 30x. The method uses advanced mathematical optimization techniques to identify and filter problematic training samples, requiring intervention on fewer than 8% of tokens.
Researchers introduce S2O, a new sparse attention method that uses online permutation and early stopping to dramatically improve AI model efficiency. The technique achieves 3.81x end-to-end speedup on Llama-3.1-8B with 128K context while maintaining accuracy.
Researchers introduce NoRA (Non-linear Rank Adaptation), a new parameter-efficient fine-tuning method that overcomes the 'linear ceiling' limitations of traditional LoRA by using SiLU gating and structural dropout. NoRA achieves superior performance at rank 64 compared to LoRA at rank 512, demonstrating significant efficiency gains in complex reasoning tasks.
New research demonstrates that AI systems trained via RLHF cannot be governed by norms due to fundamental architectural limitations in optimization-based systems. The paper argues that genuine agency requires incommensurable constraints and apophatic responsiveness, which optimization systems inherently cannot provide, making documented AI failures structural rather than correctable bugs.
Researchers identify a critical trade-off in AI model training where optimizing for Pass@k metrics (multiple attempts) degrades Pass@1 performance (single attempt). The study reveals this occurs due to gradient conflicts when the training process reweights toward low-success prompts, creating interference that hurts single-shot performance.
Researchers have introduced AIQI (Universal AI with Q-Induction), the first model-free artificial intelligence agent proven to be asymptotically optimal in general reinforcement learning. Unlike previous optimal agents like AIXI that rely on environment models, AIQI performs universal induction over distributional action-value functions, significantly expanding the diversity of known universal agents.
Researchers developed AILS-AHD, a novel approach using Large Language Models to solve the Capacitated Vehicle Routing Problem (CVRP) more efficiently. The LLM-driven method achieved new best-known solutions for 8 out of 10 instances in large-scale benchmarks, demonstrating superior performance over existing state-of-the-art solvers.
Researchers achieved breakthrough sample complexity improvements for offline reinforcement learning algorithms using f-divergence regularization, particularly for contextual bandits. The study demonstrates optimal O(ε⁻¹) sample complexity under single-policy concentrability conditions, significantly improving upon existing bounds.
Researchers introduce UniQL, a unified framework for quantizing and compressing large language models to run efficiently on mobile devices. The system achieves 4x-5.7x memory reduction and 2.7x-3.4x speed improvements while maintaining accuracy within 5% of original models.
