TileFuse: A Fused Mixed-Precision Kernel Library for Efficient Quantized LLM Inference on AMD NPUs

TileFuse is a new kernel library that enables efficient quantized large language model inference on AMD's XDNA2 NPUs by supporting industry-standard quantization formats like AWQ directly, rather than requiring model reshaping. The technology delivers up to 2x improvements in latency and energy efficiency on edge devices, making practical LLM deployment on consumer hardware substantially more viable.

TileFuse addresses a critical bottleneck in edge AI deployment: the mismatch between widely-adopted quantization standards and proprietary NPU software stacks that resist integration. AMD's XDNA2 architecture, found in Ryzen AI processors, previously lacked native support for formats like W4A16 (4-bit weights, 16-bit activations), forcing developers to choose between maintaining model compatibility or sacrificing performance. This research bridges that gap through hardware-software co-design at the kernel level.

The technical innovation lies in fusing multiple operations—unpacking, dequantization, and matrix multiplication—into single kernel flows while redesigning memory layouts and dataflow patterns specific to XDNA2's 4x8 AIE array. This eliminates intermediate memory transfers that typically plague quantized inference. The reported gains are substantial: up to 121.6% improvement over full-precision baselines for matrix-matrix operations and notably 281% for matrix-vector operations, which dominate token generation in LLM inference.

For the industry, this work signals that client NPUs can become practical acceleration targets without requiring ecosystem fragmentation around proprietary quantization schemes. This matters significantly for laptop manufacturers, software providers, and enterprise deployments seeking energy-efficient on-device AI. As regulatory pressure and privacy concerns drive preference for local inference, standardized quantization support becomes a competitive advantage.

Future implications center on whether other hardware vendors adopt similar approaches and whether TileFuse's techniques generalize to newer architectures. The research demonstrates that native support for off-the-shelf formats substantially improves developer adoption compared to proprietary alternatives.

• →TileFuse enables AMD XDNA2 NPUs to natively support AWQ-style quantization formats, eliminating forced model reshaping around hardware constraints.
• →Fused kernel design achieves up to 2.0x lower prefilling latency and 64.6% energy reduction compared to full-precision baselines on Ryzen AI laptops.
• →Matrix-vector operation performance improved 281% over full-precision, directly benefiting the token generation phase of LLM inference.
• →Hardware-software co-design optimizing weight layouts, metadata placement, and array dataflow proves essential for practical edge LLM deployment.
• →Standardized quantization support on consumer NPUs removes developer friction and accelerates adoption of on-device AI inference.