7 articles tagged with #linear-attention. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers propose Dynamic Linear Attention (DLA), a novel framework that improves how large language models process long sequences by adaptively managing memory states. DLA addresses the limitations of existing linear attention mechanisms by dynamically merging less important information while preserving critical semantic transitions, achieving superior performance across 16 datasets.
Researchers propose Kaczmarz Linear Attention (KLA), an improved algorithm for long-context language modeling that replaces empirically-learned coefficients with mathematically-derived key-norm-normalized step sizes. KLA outperforms existing linear attention baselines like Gated DeltaNet while maintaining computational efficiency and enabling stable processing of up to 65K token contexts.
MiniCPM-SALA introduces a 9B-parameter hybrid language model architecture that combines sparse and linear attention mechanisms to handle ultra-long contexts up to 1M tokens. The model achieves 3.5x faster inference than full-attention models while reducing training costs by 75% through a continual training framework that transforms existing Transformer models.
Researchers introduce Blurry Window Attention (BLA), a novel attention mechanism that addresses the quadratic complexity and memory limitations of traditional Transformer models by reconstructing sparse key-value history through Dirichlet kernel interpolation. BLA demonstrates 8x state efficiency improvements over sliding window attention while maintaining competitive performance on information retrieval tasks, positioning it as a viable alternative for long-context language modeling.
Q-Delta presents a novel approach to linear attention mechanisms in sequence modeling by integrating query-conditioned state evolution, moving beyond traditional key-value associative paradigms. The method combines efficient linear-time inference with improved performance on language modeling and long-context retrieval tasks through a hardware-optimized implementation.
Researchers propose a fast matrix multiplication-based algorithm for matrix inversion in linear attention mechanisms, achieving up to 5x speedup on neural processing units while maintaining model accuracy under both standard and low-precision inference. The method addresses a critical computational bottleneck in long-context language modeling by using truncated Neumann expansion and parallel residual correction.
Researchers reveal that Test-Time Training (TTT) with KV binding, previously understood as online meta-learning for memorization, can actually be reformulated as a learned linear attention operator. This new perspective explains previously puzzling behaviors and enables architectural simplifications and efficiency improvements.
