AINeutralarXiv – CS AI · 18h ago6/10
🧠
Contribution Weights: A Geometrical Analysis of Self-Attention Transformers
Researchers introduce Contribution Weights, a new metric for analyzing transformer attention that accounts for value vector geometry alongside attention weights. The approach more accurately identifies semantically critical tokens than traditional attention-based metrics and reveals that attention sinks actively suppress information rather than passively storing excess attention.