#kv-cache-quantization News & Analysis

2 articles tagged with #kv-cache-quantization. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

2 articles

AINeutralarXiv – CS AI · Jun 107/10

🧠

Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation

Researchers discovered that key-value cache quantization—a technique used to reduce LLM inference memory—silently degrades AI safety alignment without affecting standard performance metrics like perplexity. The study identifies the root cause as geometric vulnerability of safety features in low-dimensional activation subspaces and proposes Per-Channel Reduction (PCR), a diagnostic tool that achieves up to 97% alignment recovery without retraining.

🏢 Nvidia🏢 Perplexity

AIBullisharXiv – CS AI · May 97/10

🧠

When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon

Researchers demonstrate that int4 quantization of KV caches on Apple Silicon's unified memory architecture actually improves performance over fp16, delivering 3-8% faster inference while reducing memory usage by 3x. This inverts the traditional quality-latency tradeoff through a fused Metal kernel combining sign-randomized FFT, per-channel scaling, and int4 packing, with applications from 1B to 1.5B parameter models.

🏢 Hugging Face