When Good Enough Is Optimal: Multiplication-Only Matrix Inversion Approximation for Quantized Gated DeltaNet

Researchers propose a fast matrix multiplication-based algorithm for matrix inversion in linear attention mechanisms, achieving up to 5x speedup on neural processing units while maintaining model accuracy under both standard and low-precision inference. The method addresses a critical computational bottleneck in long-context language modeling by using truncated Neumann expansion and parallel residual correction.

This technical research tackles a fundamental performance constraint in modern large language model inference. Matrix inversion represents a sequential computational bottleneck in chunk-wise parallel linear attention mechanisms, particularly problematic on NPUs (neural processing units) where traditional forward-substitution methods fail to parallelize effectively. The proposed solution replaces sequential dependencies with a matrix multiplication-based approach, leveraging mathematical properties specific to strictly lower-triangular matrices to enable hardware-efficient parallel computation.

The innovation builds on established numerical analysis principles—Neumann series expansion and diagonal concentration properties—adapted specifically for the constraints of hardware accelerators. By truncating the expansion and applying structural masking, the authors eliminate sequential bottlenecks while introducing parallel residual correction steps that maintain numerical stability. The extension to low-bit integer arithmetic addresses a critical practical concern, as quantized inference increasingly dominates production deployments for cost and latency optimization.

The empirical validation on Qwen3.5-family models demonstrates substantial real-world impact: 5x kernel-level speedup and 20% reduction in decode-layer overhead directly translate to faster token generation and lower inference costs. Preserving accuracy across floating-point and quantized regimes indicates the method generalizes robustly. This work exemplifies how algorithmic optimization can unlock hardware utilization gains, particularly valuable as context window requirements grow and inference becomes computationally dominant relative to training in production systems.

Future relevance depends on adoption across LLM frameworks and NPU manufacturers' support for the specific optimization patterns. The technique's applicability to other structured matrix operations in attention mechanisms could extend its impact.

• →Proposes a MatMul-based algorithm achieving 5x speedup in matrix inversion for linear attention on NPUs
• →Uses truncated Neumann expansion with structural masking to eliminate sequential dependencies and enable parallelization
• →Extends to low-bit integer quantization by mitigating dynamic range expansion in repeated matrix operations
• →Demonstrates 20% reduction in decode-layer computational overhead while preserving model accuracy
• →Addresses critical bottleneck in long-context modeling as context windows and inference demands grow