#hybrid-attention News & Analysis

2 articles tagged with #hybrid-attention. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

2 articles

AIBullisharXiv – CS AI · Mar 37/102

🧠

MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling

MiniCPM-SALA introduces a 9B-parameter hybrid language model architecture that combines sparse and linear attention mechanisms to handle ultra-long contexts up to 1M tokens. The model achieves 3.5x faster inference than full-attention models while reducing training costs by 75% through a continual training framework that transforms existing Transformer models.

AIBullisharXiv – CS AI · Jun 26/10

🧠

FLARE: Diffusion for Hybrid Language Model

Researchers introduce FLARE, a conversion framework that enables large language models with hybrid attention mechanisms to function as both autoregressive and diffusion models, addressing a key limitation in parallel decoding while maintaining model capability. The approach demonstrates competitive performance with existing diffusion language models while delivering throughput gains in concurrent serving scenarios.