Toeplitz MLP Mixers are Low Complexity, Information-Rich Sequence Models

Researchers introduce Toeplitz MLP Mixer (TMM), a transformer alternative that replaces attention mechanisms with triangular-masked Toeplitz matrix multiplication, achieving O(dn log n) training complexity and O(dn) inference complexity. TMMs demonstrate superior training efficiency, information retention, and in-context learning performance compared to existing sub-quadratic architectures.

The computational constraints of transformer-based language models have become a critical bottleneck as model sizes and sequence lengths increase. The quadratic complexity of attention mechanisms creates substantial challenges for both training and inference, particularly at scale. The introduction of Toeplitz MLP Mixers represents a meaningful attempt to address this fundamental limitation through architectural innovation rather than incremental optimization.

TMMs achieve their efficiency gains by replacing the computationally expensive attention mechanism with structured matrix operations that preserve causal masking while reducing complexity. The triangular-masked Toeplitz structure elegantly maintains the sequential dependencies necessary for language modeling while enabling faster computation. What distinguishes this approach is its simplicity—unlike other sub-quadratic alternatives that employ sophisticated input modulation or state maintenance mechanisms, TMMs accomplish comparable or superior results through structural efficiency alone.

The empirical advantages revealed in the research carry significant implications for model developers and researchers. Superior information retention directly translates to improved copying ability and in-context learning, suggesting TMMs may achieve better task performance with equivalent or reduced computational resources. This efficiency-to-performance ratio addresses a practical pain point in model development and deployment, where computational budgets remain finite despite increasing performance demands.

The theoretical analysis through operator index theory provides additional validation, demonstrating that trained Toeplitz layers exhibit invertibility properties despite operating in non-invertible settings. This counterintuitive finding suggests the architecture naturally learns robust representations. For practitioners, TMMs offer a compelling alternative that reduces memory footprint and training time while maintaining or exceeding performance on critical benchmarks, potentially shifting architectural preferences in the research community.

• →Toeplitz MLP Mixers reduce transformer attention complexity from quadratic to O(dn log n) during training and O(dn) at inference
• →TMMs achieve superior training efficiency and retain more input information than comparable sub-quadratic architectures
• →The architecture demonstrates improved in-context learning and information retrieval benchmark performance without sophisticated state mechanisms
• →Operator index theory analysis reveals trained Toeplitz layers develop invertibility properties, providing theoretical validation of learned representations
• →Simpler architectural design with lower computational overhead positions TMMs as a practical alternative for resource-constrained model development