Why Smaller Is Slower? Dimensional Misalignment in Compressed LLMs

Researchers identify dimensional misalignment as a critical bottleneck in compressed large language models, where parameter reduction fails to improve GPU performance due to hardware-incompatible tensor dimensions. They propose GAC (GPU-Aligned Compression), a new optimization method that achieves up to 1.5× speedup while maintaining model quality by ensuring hardware-friendly dimensions.

The efficiency of large language models depends not just on parameter counts but on how those parameters align with GPU hardware capabilities. Compressed LLMs paradoxically run slower than their uncompressed counterparts despite using fewer parameters—a counterintuitive finding that challenges common assumptions about model optimization. The research traces this problem across three architectural layers: the software framework, numerical libraries, and GPU hardware, revealing that popular compression techniques like activation-aware singular value decomposition produce tensor dimensions that GPU execution stacks cannot efficiently process.

This work addresses a growing pain in the AI acceleration space. As organizations race to deploy smaller, more efficient models, compression has emerged as a standard practice. However, the disconnect between compression optimization objectives and hardware execution constraints has created a hidden performance tax. For developers and organizations deploying compressed models in production, this finding has immediate relevance—parameter reduction alone provides no guarantee of inference speedup.

The proposed GAC solution reformulates compression as a constrained optimization problem, treating GPU alignment requirements as first-class constraints rather than afterthoughts. By converting any dimension-reducing compressor into a hardware-aware variant through multi-choice knapsack optimization, GAC maintains the same parameter budget while achieving full hardware alignment. This pragmatic approach bridges the gap between model compression theory and GPU execution reality, enabling developers to actually realize the latency benefits they expect from model compression.

The industry implication extends beyond academic interest. Production ML systems can now achieve genuine speedups from compression, potentially reducing inference costs and energy consumption simultaneously. This work opens a pathway for more efficient edge deployment and cost-effective cloud inference.

• →Compressed LLMs often run slower than uncompressed versions due to GPU-incompatible tensor dimensions, a phenomenon called dimensional misalignment.
• →Popular compression techniques like ASVD can reduce parameters by 15% while achieving zero speedup because 95% of dimensions misalign with GPU hardware.
• →GAC (GPU-Aligned Compression) wraps existing compressors and re-optimizes dimensions for hardware alignment via multi-choice knapsack optimization.
• →GAC achieves 100% hardware alignment and recovers up to 1.5× speedup on Llama-3-8B while preserving model quality.
• →This work bridges the gap between compression optimization theory and practical GPU execution, enabling genuine inference speedups in production systems.