y0news
AnalyticsDigestsSourcesTopicsRSSAICrypto

#self-attention News & Analysis

7 articles tagged with #self-attention. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

7 articles
AINeutralarXiv – CS AI · Jun 117/10
🧠

From Architecture to Output: Structural Origins of Hallucination in Large Language Models and the Amplifying Role of Data

Researchers identify three core architectural mechanisms in large language models that systematically produce hallucinations: self-attention's statistical confusion of entities, maximum likelihood training that rewards plausible-sounding falsehoods, and autoregressive decoding that cascades errors forward. Dataset quality issues amplify rather than originate these failures, suggesting that fixing hallucinations requires architectural redesign, not just better training data.

AINeutralarXiv – CS AI · Apr 147/10
🧠

A Mathematical Explanation of Transformers

Researchers propose a novel mathematical framework interpreting Transformers as discretized integro-differential equations, revealing self-attention as a non-local integral operator and layer normalization as time-dependent projection. This theoretical foundation bridges deep learning architectures with continuous mathematical modeling, offering new insights for architecture design and interpretability.

AIBullisharXiv – CS AI · Mar 57/10
🧠

Quantum-Inspired Self-Attention in a Large Language Model

Researchers developed a quantum-inspired self-attention (QISA) mechanism and integrated it into GPT-1's language modeling pipeline, marking the first such integration in autoregressive language models. The QISA mechanism demonstrated significant performance improvements over standard self-attention, achieving 15.5x better character error rate and 13x better cross-entropy loss with only 2.6x longer inference time.

AIBullisharXiv – CS AI · Mar 46/103
🧠

On the Expressive Power of Transformers for Maxout Networks and Continuous Piecewise Linear Functions

Researchers establish theoretical foundations for Transformer networks' expressive power by connecting them to maxout networks and continuous piecewise linear functions. The study proves Transformers inherit universal approximation capabilities of ReLU networks while revealing that self-attention layers implement max-type operations and feedforward layers perform token-wise affine transformations.

AINeutralarXiv – CS AI · Jun 26/10
🧠

Soft-NBCE: Entropy-Weighted Chunk Fusion for Long-Context

Researchers introduce Soft-NBCE, an improved method for processing ultra-long text contexts in large language models by replacing discrete chunk selection with weighted chunk fusion. The approach demonstrates measurable improvements on multi-hop reasoning tasks while maintaining efficient memory usage, addressing a critical bottleneck in LLM inference.