Forget Attention: Importance-Aware Attention Is All You Need

Researchers propose SISA (SSM-Informed Softmax Attention), a hybrid architecture that integrates state space model importance signals directly into transformer attention mechanisms at the score level. The approach achieves superior performance on language modeling benchmarks, particularly excelling at long-context retrieval tasks while maintaining computational efficiency through standard operations.

SISA addresses a fundamental limitation in current hybrid language models: the inability to combine transformers' global context awareness with state space models' sequential importance prioritization. While previous hybrids like Jamba and Hymba keep these mechanisms segregated, SISA fuses them within the attention computation itself by incorporating SSM-derived importance weights into attention scores, implemented elegantly through standard SDPA operations without custom kernels or recurrent state management.

The technical innovation enables a cleaner architectural design compared to existing block-level or head-level fusion approaches. By operating at the score level, SISA creates a more integrated system where importance signals actively influence which tokens receive attention, rather than treating attention and importance as parallel but separate pathways.

Benchmark results demonstrate concrete advantages: at 152M parameters, SISA reaches 17.3% on LAMBADA-greedy (exceeding pure transformer at 13.9% and Mamba-3 at 15.5%), while achieving 100% accuracy on the "Needle in a Haystack" test 7x faster than transformers. The approach maintains computational compatibility with standard deep learning libraries, reducing implementation friction for widespread adoption. At larger scales (369M parameters), performance variations emerge, suggesting the method's benefits vary with model capacity.

For the AI research community, SISA opens a new design dimension for hybrid architectures that emphasizes score-level fusion over structural separation. This could accelerate exploration of more tightly integrated SSM-attention combinations. The work validates that importance-aware mechanisms complement rather than contradict global attention, potentially influencing how future large language models balance long-context capabilities with computational efficiency.

• →SISA introduces score-level fusion as a third design paradigm for SSM-attention hybrids beyond existing block-level and head-level approaches.
• →The method achieves 17.3% LAMBADA accuracy at 152M parameters, outperforming standard transformers and Mamba-3 on next-token prediction.
• →SISA reaches perfect long-context retrieval (NIAH 100%) 7x faster than transformers while using only standard SDPA operations.
• →The approach eliminates custom kernels and recurrent state requirements by augmenting query/key vectors with importance information.
• →Performance gains diminish at larger model scales (369M parameters), suggesting benefits are most pronounced for smaller language models.