Do Transformers Need Three Projections? Systematic Study of QKV Variants

Researchers systematically evaluate whether transformer models require three separate QKV projections, discovering that shared projection variants perform comparably while reducing computational overhead. The Q-K=V configuration achieves 50% KV cache reduction with minimal performance loss and combines effectively with existing optimization techniques like MQA to enable practical on-device deployment.

This research addresses a fundamental architectural assumption in modern transformers that has remained largely unquestioned since their introduction. The study challenges the necessity of maintaining three independent projection matrices for queries, keys, and values—a design choice that adds computational and memory overhead without clear justification. By systematically evaluating projection sharing constraints across diverse tasks, the researchers provide empirical evidence that this standard formulation contains redundancy that can be exploited without sacrificing model quality.

The work builds on broader trends in neural network efficiency optimization, where researchers increasingly scrutinize inherited architectural patterns. Prior work on head sharing (GQA/MQA) demonstrated significant inference benefits; this research shows projection sharing operates orthogonally, creating multiplicative memory savings. The finding that Q-K=V preserves performance because keys and values occupy similar representational spaces reveals something fundamental about attention mechanisms—they operate in inherently low-rank regimes where redundancy is exploitable.

For practitioners deploying transformers in resource-constrained environments, these results have direct quantifiable benefits. The 96.9% KV cache reduction achieved by combining Q-K=V with MQA makes large language models practically viable on edge devices, a critical capability as inference moves from cloud infrastructure to edge hardware. This impacts mobile AI applications, IoT deployments, and reduces latency-critical inference costs. The public code release enables immediate adoption across production systems.

The research establishes projection sharing as an underexplored efficiency lever within the broader weight-tying ecosystem. As model compression remains central to AI deployment economics, systematic characterization of architectural redundancies provides developers with validated optimization strategies beyond conventional pruning and quantization approaches.

• →Q-K=V projection sharing achieves 50% KV cache reduction with only 3.1% perplexity degradation in large language models
• →Combining projection sharing with MQA enables 96.9% total cache reduction, making on-device inference practically feasible
• →Three independent projections contain exploitable redundancy because keys and values occupy similar representational spaces
• →Projection sharing is orthogonal to existing head-sharing techniques, creating multiplicative memory optimization benefits
• →Systematic empirical evaluation reveals that transformer's standard QKV formulation includes unexamined architectural assumptions