On the Expressive Power of Transformers for Maxout Networks and Continuous Piecewise Linear Functions

Researchers establish theoretical foundations for Transformer networks' expressive power by connecting them to maxout networks and continuous piecewise linear functions. The study proves Transformers inherit universal approximation capabilities of ReLU networks while revealing that self-attention layers implement max-type operations and feedforward layers perform token-wise affine transformations.

• →Transformer networks can explicitly approximate maxout networks while maintaining comparable model complexity.
• →Transformers inherit the universal approximation capability of ReLU networks under similar complexity constraints.
• →The expressivity of Transformers can be quantified by the number of linear regions, which grows exponentially with depth.
• →Self-attention layers implement max-type operations while feedforward layers realize token-wise affine transformations.
• →The research establishes a theoretical bridge between approximation theory for feedforward neural networks and Transformer architectures.