AINeutralarXiv – CS AI · Jun 236/10
🧠
Massive Activations Are Architecturally Robust: A Controlled Scratch/Commitment Residual Stream Test
Researchers tested whether massive activations in transformer neural networks are architectural artifacts or functionally necessary by creating a specialized architecture (Ledger Residuals) that separates the residual stream into scratch and protected channels. The model rebuilt the massive activation pattern in the protected channel regardless, suggesting these outliers serve a functional purpose rather than being removable byproducts of design constraints.