Pushing the Limits of Block Rotations in Post-Training Quantization

Researchers present PeRQ, a post-training quantization method that uses permutations to optimize block rotations for neural network compression. The approach recovers up to 90% of full-vector rotation performance when quantizing large language models to INT4, significantly outperforming existing block rotation methods.

Post-training quantization enables efficient deployment of large language models by reducing precision without retraining, making inference faster and cheaper. Block rotations have emerged as a practical technique to handle outlier values that complicate quantization, but their effectiveness depends heavily on how input data distributes across blocks. PeRQ advances this technique by identifying a fundamental constraint: outlier suppression improves when activation mass distributes evenly across blocks rather than concentrating in specific regions.

The research addresses a critical gap in understanding quantization mechanics. Previous block rotation methods achieved only 46% of full-vector rotation performance on Llama3 1B with INT4 quantization and block size 16. PeRQ's permutation-based approach redistributes this mass before rotation, then embeds the permutations directly into model weights to eliminate runtime overhead. This architectural insight is significant because it solves the practical deployment challenge that has limited adoption of rotation-based quantization.

For the AI industry, this directly impacts model efficiency at scale. Companies deploying large language models can now achieve better accuracy-to-size tradeoffs, reducing computational requirements for inference. The method's compatibility with transformer architectures and elimination of deployment overhead removes barriers to practical implementation. The 90% recovery rate represents meaningful improvement over existing PTQ methods, particularly for edge deployments and resource-constrained environments where INT4 quantization is essential.

Future development should focus on extending these principles to other quantization schemes and exploring whether similar permutation strategies benefit other neural architecture patterns beyond transformers.

• →PeRQ achieves 90% of full-vector rotation performance using permutations, versus 46% without them, on Llama3 1B INT4 quantization.
• →Block rotation effectiveness is fundamentally limited by how activation mass distributes across blocks, solved through pre-rotation permutations.
• →Permutations merge into model weights during deployment, eliminating inference overhead and enabling practical production use.
• →The method works across all block sizes and consistently improves accuracy compared to standard block rotation approaches.
• →Research identifies permutation-equivariant regions in transformers, bridging theoretical optimization with practical architectural constraints.