26 articles tagged with #llm-compression. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers propose UniRank, a new method for efficiently allocating ranks in low-rank decomposition of large language models by scoring components via local singular energy and global functional importance. The approach achieves up to 50% perplexity reduction compared to baseline methods without additional fine-tuning, addressing a key bottleneck in LLM compression.
Researchers introduce LC-QAT, a novel 2-bit quantization method for large language models that combines vector quantization with learnable affine mappings to achieve superior compression with minimal training data. The approach outperforms existing quantization-aware training methods while requiring only 0.1-10% of typical training data, advancing the practical deployment of extremely low-bit LLMs.
Researchers introduce an end-to-end framework for compressing Large Language Models through joint structural pruning and mixed-precision quantization that optimizes global error propagation rather than layer-wise errors. The approach demonstrates significant performance improvements at ultra-low bit precisions (1-3 bits), reducing perplexity by up to 21% compared to existing methods.
Researchers introduce LLMCodec, a novel compression method that adapts video codecs like VVC/H.266 to efficiently compress large language models. The approach achieves significant improvements over existing quantization methods, reducing perplexity by 1.5x on LLaMA-3-8B at 2-bit precision while improving downstream task accuracy by 21%.
QuBLAST is a new post-training quantization method that compresses large language models by 40-45% while maintaining performance, using block-level mixed-precision quantization and activation scaling to address computational and memory constraints in LLM deployment.
BitsMoE introduces a spectral-energy-guided quantization framework for compressing Mixture-of-Experts large language models, achieving significant improvements in the ultra-low-bit regime. The method uses SVD decomposition to intelligently allocate bits across expert weights, delivering 27.83 percentage point accuracy improvements over existing approaches at 2-bit quantization while accelerating inference speed by 1.76× on Qwen models.
Researchers introduce SubFit, a post-training compression method for Large Language Models that operates at the submodule level rather than full-layer granularity, achieving superior perplexity-accuracy trade-offs. The approach selects non-contiguous Attention and FeedForward submodules with individual fitted residual bypasses, delivering 84.6% downstream accuracy retention at 25% sparsity compared to 81.6% for existing methods.
Researchers propose Locality-Aware Redundancy Pruning (LoRP), a training-free method for compressing large language models by removing redundant layers based on representational similarity patterns. The framework uses a Representation Locality Score to identify and prune depth-wise redundancy more effectively than existing approaches, improving both perplexity and downstream task performance across multiple LLM architectures.
A comprehensive empirical study reveals that weight pruning—a technique for compressing large language models for edge devices—paradoxically amplifies bias while preserving performance metrics. The research shows activation-aware pruning methods maintain perplexity but increase stereotype reliance by up to 84%, suggesting current evaluation methods fail to detect fairness degradation in compressed models.
Researchers propose SARQC, a new post-training quantization framework for large language models that adds saliency-aware regularization to prevent quantized weights from drifting too far from original values. The method improves generalization performance across dense and mixture-of-experts LLMs without increasing inference costs.
Researchers demonstrate that int4 quantization of KV caches on Apple Silicon's unified memory architecture actually improves performance over fp16, delivering 3-8% faster inference while reducing memory usage by 3x. This inverts the traditional quality-latency tradeoff through a fused Metal kernel combining sign-randomized FFT, per-channel scaling, and int4 packing, with applications from 1B to 1.5B parameter models.
Researchers introduce FASQ, a calibration-free compression framework for large language models that uses product quantization to achieve flexible compression ratios between 27-49% of original model size. The method outperforms existing quantization approaches like GPTQ and AWQ while enabling faster inference than FP16 on consumer GPUs through custom CUDA kernels.
Researchers identify dimensional misalignment as a critical bottleneck in compressed large language models, where parameter reduction fails to improve GPU performance due to hardware-incompatible tensor dimensions. They propose GAC (GPU-Aligned Compression), a new optimization method that achieves up to 1.5× speedup while maintaining model quality by ensuring hardware-friendly dimensions.
Research demonstrates that layer pruning—a compression technique for large language models—effectively reduces model size while maintaining classification performance, but critically fails to preserve generative reasoning capabilities like arithmetic and code generation. Even with extensive post-training on 400B tokens, models cannot recover lost reasoning abilities, revealing fundamental limitations in current compression approaches.
Researchers propose SoLA, a training-free compression method for large language models that combines soft activation sparsity and low-rank decomposition. The method achieves significant compression while improving performance, demonstrating 30% compression on LLaMA-2-70B with reduced perplexity from 6.95 to 4.44 and 10% better downstream task accuracy.
Researchers propose ERC-SVD, a new compression method for large language models that uses error-controlled singular value decomposition to reduce model size while maintaining performance. The method addresses truncation loss and error propagation issues in existing SVD-based compression techniques by leveraging residual matrices and selectively compressing only the last few layers.
Researchers analyzed compression effects on large reasoning models (LRMs) through quantization, distillation, and pruning methods. They found that dynamically quantized 2.51-bit models maintain near-original performance, while identifying critical weight components and showing that protecting just 2% of excessively compressed weights can improve accuracy by 6.57%.
Researchers introduce EPTS, a new framework for compressing large language models that enables a single optimized model to perform efficiently across multiple sparsity levels, eliminating the need for separate optimization for each deployment scenario. This approach combines Multi-Sparsity Hierarchy LoRA and a Feature Mixer mechanism to maintain performance while reducing computational requirements.
Researchers conducted a systematic empirical study evaluating quantization methods for OpenPangu language models on Huawei Ascend NPUs, finding that 8-bit weight-only quantization is lossless while 4-bit quantization remains practical for larger models but degrades performance on reasoning tasks in smaller models. The study reveals that extreme low-bit compression (2-bit and binary) remains fundamentally challenging, with most configurations collapsing to near-random behavior.
Researchers propose EinSort, an adaptive tensorization method that uses index ordering to identify and compress low-rank structures in large language models, demonstrating improved results for weight and KV-cache compression compared to existing approaches.
Researchers propose FAIR-Calib, a novel post-training quantization framework designed to address instability issues in Diffusion Large Language Models (dLLMs) where early token decisions become permanently locked despite remaining fragile. The two-stage method uses frontier-aware reweighting to protect critical decision points during model compression, demonstrating improved performance over existing quantization baselines.
Researchers introduce a differentiable Neural Architecture Search framework that jointly optimizes LLM architecture and mixed-precision quantization, achieving 1.4x faster inference speeds or 6% higher accuracy compared to sequential optimization approaches. This compression technique addresses the critical challenge of deploying large language models on edge devices without requiring extensive GPU training.
Researchers benchmark 12 LLMs under compression to evaluate whether quantization and pruning preserve uncertainty quantification alongside accuracy. The study reveals compression frequently decouples accuracy from uncertainty reliability, with smaller models absorbing compression-induced uncertainty poorly, suggesting current accuracy-only evaluation standards are insufficient for deployment readiness.
Researchers present a method for aggressively pruning expert modules from mixture-of-experts large language models to create specialized translation systems. The approach removes up to 90% of experts with minimal performance degradation, demonstrating that translation tasks require only a fraction of a full LLM's parameters, enabling substantial model compression.
Researchers have developed KDFlow, a new framework for compressing large language models that achieves 1.44x to 6.36x faster training speeds compared to existing knowledge distillation methods. The framework uses a decoupled architecture that optimizes both training and inference efficiency while reducing communication costs through innovative data transfer techniques.
