Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference on Edge Devices

Researchers introduce Vec-LUT, a novel vector-based lookup table technique that dramatically improves ultra-low-bit LLM inference on edge devices by addressing memory bandwidth underutilization. The method achieves up to 4.2x performance improvements over existing approaches, enabling faster LLM execution on CPUs than specialized NPUs.

Vec-LUT addresses a critical bottleneck in edge-device LLM deployment where the race to ultra-low quantization (1.58-bit to 4-bit) has outpaced inference efficiency. While LUT-based inference has proven effective for reducing computational complexity, the scalar lookup paradigm creates scattered memory access patterns that waste valuable bandwidth during parallel token processing—essential for prefilling and batch inference scenarios. This inefficiency becomes pronounced precisely when edge deployment demands it most: handling multiple tokens simultaneously on resource-constrained hardware.

The solution elegantly reframes the problem through vector parallelism, constructing unified lookup tables across multiple tokens and performing single 1→N lookups instead of repetitive scalar operations. The technical innovations—Vector LUT-Centric Tensor Layout and Cache-Aware Streamed Lookup—demonstrate practical engineering that bridges algorithmic improvement with hardware constraints. Testing across five edge device types and three LLM architectures validates broad applicability.

For the edge AI market, this matters substantially. CPUs now potentially outperform dedicated NPUs for quantized inference, shifting deployment economics and opening edge AI to billions of existing devices. Developers gain actionable tools immediately through llama.cpp integration, enabling production optimization without waiting for hardware refreshes. The implications extend to privacy-preserving on-device AI, reducing reliance on cloud inference for latency-sensitive applications.

The work signals maturation in edge LLM optimization, where incremental algorithmic improvements now yield substantial practical gains. Future developments may focus on dynamic quantization levels or heterogeneous inference strategies leveraging CPU advantages in specific scenarios.

• →Vec-LUT achieves 4.2x performance improvement for ultra-low-bit LLM inference on edge CPUs through vector-based lookup optimization.
• →Memory bandwidth utilization becomes the primary bottleneck in parallel token processing, which Vec-LUT addresses through unified lookup tables.
• →CPUs now outperform NPUs for quantized LLM inference, reshaping edge device deployment economics.
• →Implementation available in llama.cpp enables immediate adoption for developers building on-device AI applications.
• →Technique applies across diverse edge hardware and LLM architectures, suggesting broad industry relevance.