#routing-networks News & Analysis

2 articles tagged with #routing-networks. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

2 articles

AINeutralarXiv – CS AI · Jun 237/10

🧠

All Routes Lead to Collapse

Researchers demonstrate that attention sinks, representation collapse, and norm stratification—previously thought to be transformer-specific problems—are universal behaviors of content-based routing systems with mismatched metrics. The study reveals this collapse pattern occurs across diverse architectures including softmax attention, graph attention, state-space models, and recurrent mixers, suggesting the issue stems from fundamental routing mechanics rather than transformer design.

AIBullisharXiv – CS AI · Jun 97/10

🧠

Chiaroscuro Attention: Spending Compute in the Dark

Researchers introduce CHIAR-Former, a hybrid transformer that routes tokens to different operators (DCT spectral mixing, RBF kernel mixing, or full self-attention) based on spectral entropy. The DCT+Attention variant achieves 45% better perplexity than standard attention on WikiText-103 while using 62.5% fewer attention operations, demonstrating significant computational efficiency gains for large-scale language models.