144 articles tagged with #quantization. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers reproduced and analyzed severe accuracy degradation in BERT transformer models when applying post-training quantization, showing validation accuracy drops from 89.66% to 54.33%. The study found that structured activation outliers intensify with model depth, with mixed precision quantization being the most effective mitigation strategy.
Researchers successfully developed Bielik-Q2-Sharp, the first systematic evaluation of extreme 2-bit quantization for Polish language models, achieving near-baseline performance while significantly reducing model size. The study compared six quantization methods on an 11B parameter model, with the best variant maintaining 71.92% benchmark performance versus 72.07% baseline at just 3.26 GB.
Researchers developed a training method for large-scale Mixture-of-Experts (MoE) models using FP4 precision on Hopper GPUs without native 4-bit support. The technique achieves 14.8% memory reduction and 12.5% throughput improvement for 671B parameter models by using FP4 for activations while keeping core computations in FP8.
Researchers prove that the GPTQ neural network quantization algorithm is mathematically equivalent to Babai's nearest-plane algorithm for solving lattice problems. The work establishes a connection between neural network quantization and lattice geometry, suggesting potential improvements through lattice basis reduction techniques.
Researchers propose SUN (Shared Use of Next-token Prediction), a novel approach for multi-LLM serving that enables cross-model sharing of decode execution by decomposing transformers into separate prefill and decode modules. The system achieves up to 2.0x throughput improvement per GPU while maintaining accuracy comparable to full fine-tuning, with a quantized version (QSUN) providing additional 45% speedup.
Researchers have developed SageBwd, a trainable INT8 attention mechanism that can match full-precision attention performance during pre-training while quantizing six of seven attention matrix multiplications. The study identifies key factors for stable training including QK-norm requirements and the impact of tokens per step on quantization errors.
Researchers introduce the first theoretical framework analyzing convergence of adaptive optimizers like Adam and Muon under floating-point quantization in low-precision training. The study shows these algorithms maintain near full-precision performance when mantissa length scales logarithmically with iterations, with Muon proving more robust than Adam to quantization errors.
ButterflyMoE introduces a breakthrough approach to reduce memory requirements for AI expert models by 150× through geometric parameterization instead of storing independent weight matrices. The method uses shared ternary prototypes with learned rotations to achieve sub-linear memory scaling, enabling deployment of multiple experts on edge devices.
Researchers developed HierarchicalPrune, a compression framework that reduces large-scale text-to-image diffusion models' memory footprint by 77.5-80.4% and latency by 27.9-38.0% while maintaining image quality. The technique enables billion-parameter AI models to run efficiently on resource-constrained devices through hierarchical pruning and knowledge distillation.
Researchers introduce UniQL, a unified framework for quantizing and compressing large language models to run efficiently on mobile devices. The system achieves 4x-5.7x memory reduction and 2.7x-3.4x speed improvements while maintaining accuracy within 5% of original models.
Researchers developed a new approach to quantization-aware training (QAT) that optimizes compute allocation between full-precision and quantized training phases. They discovered that contrary to previous findings, the optimal ratio of QAT to full-precision training increases with total compute budget, and derived scaling laws to predict optimal configurations across different model sizes and bit widths.
FlashOptim introduces memory optimization techniques that reduce AI training memory requirements by over 50% per parameter while maintaining model quality. The suite reduces AdamW memory usage from 16 bytes to 7 bytes per parameter through improved master weight splitting and 8-bit optimizer state quantization.
Tencent Hunyuan team introduces AngelSlim, a comprehensive toolkit for large model compression featuring quantization, speculative decoding, and pruning techniques. The toolkit includes the first industrially viable 2-bit large model (HY-1.8B-int2) and achieves 1.8x to 2.0x throughput gains while maintaining output quality.
Researchers developed a runtime-reconfigurable bitwise systolic array architecture for multi-precision quantized neural networks on FPGA hardware accelerators. The system achieves 1.3-3.6x speedup on mixed-precision models while supporting higher clock frequencies up to 250MHz, addressing the trade-off between hardware efficiency and inference accuracy.
The article discusses techniques for fine-tuning large language models (LLMs) to achieve extreme quantization down to 1.58 bits, making the process more accessible and efficient. This represents a significant advancement in model compression technology that could reduce computational requirements and costs for AI deployment.
The article discusses advances in making Large Language Models (LLMs) more accessible through bitsandbytes library, 4-bit quantization techniques, and QLoRA (Quantized Low-Rank Adaptation). These technologies enable running and fine-tuning large AI models on consumer hardware with significantly reduced memory requirements.
Researchers introduce HiReLC, a hierarchical reinforcement learning framework that automates the joint compression of neural networks through pruning and quantization. The system achieves 5.99-6.72x compression ratios across Vision Transformers and CNNs with minimal accuracy loss, using a two-level agent architecture guided by Fisher Information sensitivity estimates.
Researchers conducted a systematic empirical study evaluating quantization methods for OpenPangu language models on Huawei Ascend NPUs, finding that 8-bit weight-only quantization is lossless while 4-bit quantization remains practical for larger models but degrades performance on reasoning tasks in smaller models. The study reveals that extreme low-bit compression (2-bit and binary) remains fundamentally challenging, with most configurations collapsing to near-random behavior.
Researchers introduce ScalePredictor, a dynamic quantization framework that optimizes Vision Transformer deployment on edge devices by learning instance-aware quantization scales. The method leverages correlations between shallow-layer activation distributions and deeper-layer optimal scales, achieving superior accuracy-efficiency trade-offs compared to existing post-training quantization approaches.
Researchers introduce SCENIC, a neural framework designed to optimize language models for edge IoT devices by enabling them to convert natural language commands into structured smart-home instructions. The system achieves 99% accuracy on benchmarks while reducing model size by 25% through pruning and quantization, addressing the practical challenge of deploying AI on memory-constrained devices.
Researchers introduce MINCE, a novel method that significantly reduces the computational cost of evaluating large language models by intelligently shrinking benchmark datasets. Using Monte Carlo simulation with minimal calibration models, MINCE achieves 54-89% dataset size reductions while maintaining accuracy within acceptable drift thresholds, enabling 2.7-8.1x faster GPU evaluations.
Researchers introduce memory optimization techniques for fine-tuning Large Language Models using LoRA on resource-constrained devices, achieving up to 28× peak memory reduction through quantization, efficient checkpointing, and token approximation methods. The work enables private model personalization on consumer hardware without compromising model quality.
Researchers introduce UltraQuant, a 4-bit key-value cache compression technique optimized for long-context AI agents that need to process multiple conversation turns efficiently. The method achieves 3.47x faster response times in cache-pressured scenarios and 1.63x higher throughput compared to standard FP8 approaches, with practical optimizations for AMD GPU deployment.
Researchers propose q-PDGD, a quantized stochastic primal-dual optimization method for distributed systems with limited communication bandwidth. The approach achieves linear convergence under relaxed geometric conditions and matches centralized stochastic optimization rates while reducing communication overhead through quantization.
Researchers developed a federated learning system for ECG anomaly detection that simultaneously achieves GDPR/HIPAA compliance, real-time edge device performance, and clinical-grade detection accuracy across non-uniform hospital data. The system combines differential privacy, quantization, and federated averaging to enable privacy-preserving cardiac monitoring on resource-constrained hardware like Raspberry Pi 4.
