Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection

Researchers propose asymmetric transformer attention where keys use fewer dimensions than queries and values, achieving 75% key cache reduction with minimal quality loss. The technique enables 60% more concurrent users for large language models by saving 25GB of KV cache per user for 7B parameter models.

• →Asymmetric attention reduces key dimensionality to 1/4 of model dimension with only 4.3% perplexity increase on language modeling tasks
• →SVD compression followed by lightweight fine-tuning achieves 75% key cache savings at less than 2% quality cost for existing models
• →The approach enables approximately 60% more concurrent users on the same GPU hardware for large language model serving
• →Keys are significantly more compressible than queries, requiring only O(log N) dimensions to distinguish among N patterns
• →The technique was validated across multiple model sizes from 125M to 7.2B parameters with consistent results