Sparsity Moves Computation: How FFN Architecture Reshapes Attention in Small Transformers

Researchers studying one-layer Transformers discovered that architectural choices in feedforward networks (FFNs)—particularly sparse mixture-of-experts (MoE) routing—fundamentally reshape how attention mechanisms learn to compute, with sparsity rather than learned specialization driving this computational redistribution.

This research reveals a critical interdependency in Transformer architecture that challenges conventional assumptions about modular design. The study examined how different FFN architectures (dense, gated linear units, mixture-of-experts) influence the learning dynamics of attention mechanisms when models tackle tasks requiring sequential reasoning, such as digit addition with carry. The key finding demonstrates that sparse routing architectures force computation to migrate from the FFN to the attention layer, a phenomenon driven primarily by reduced capacity and sparse partitioning rather than the router's learned specialization patterns.

The research builds on growing interest in understanding Transformer internals as the field pushes toward more efficient and interpretable models. Previous work focused on individual components in isolation; this study reveals that local architectural decisions propagate globally across the model. The discovery that frozen random routing nearly matches learned routing has profound implications for mixture-of-experts systems, suggesting significant computational overhead in router learning may be unnecessary.

For the AI development community, these findings impact how engineers design efficient language models and specialized systems. Understanding that FFN sparsity inherently redistributes computation provides concrete guidance for architecture selection when optimizing for inference speed or parameter efficiency. The work on GLU-style gating also demonstrates that multiplicative mechanisms obscure neuron-level interpretability while preserving task performance, complicating mechanistic interpretability research.

Future investigation should explore whether these redistribution patterns scale to larger models and whether they persist across diverse task distributions beyond arithmetic reasoning, potentially informing the design of next-generation efficient Transformers.

• →Sparse MoE routing shifts computational burden from FFN to attention mechanisms, driven by architectural sparsity rather than learned router specialization.
• →Frozen random expert routing nearly matches learned routing performance, suggesting router learning may represent unnecessary computational overhead.
• →GLU-style multiplicative gating redistributes task-relevant information into distributed subspaces, reducing neuron-level interpretability effectiveness.
• →FFN architectural choices produce nonlocal consequences throughout the entire Transformer, not just within the block itself.
• →These findings provide concrete design guidance for building efficient Transformers with predictable computational behavior.