When Attention Beats Fourier: Multi-Scale Transformers for PDE Solving on Irregular Domains

Researchers introduce Multi-Scale Attention Transformer (MSAT), a deep learning architecture that outperforms Fourier-based neural operators for solving PDEs on irregular domains. The model achieves 3.7x better accuracy than FNO on complex geometry problems while running 3,500x faster than competing approaches, with theoretical bounds explaining when attention mechanisms beat frequency-domain methods.

This research addresses a fundamental question in scientific machine learning: which neural architectures best approximate solutions to partial differential equations across different problem classes. The work demonstrates that transformer-based attention mechanisms significantly outperform Fourier neural operators (FNO) on problems with complex, irregular geometries—achieving 0.0101 relative L2 error on Heat2D-CG versus FNO's 0.037. The computational efficiency gain is equally striking, with MSAT requiring 34 seconds for inference compared to Mamba-NO's 120,812 seconds, suggesting practical viability for real-time scientific computing applications.

The research positions itself within a maturing landscape where neural operators have emerged as alternatives to physics-informed neural networks (PINNs). Where FNO assumes regular grids enabling efficient Fourier transforms, irregular domains break this assumption and reveal attention's advantage: learned token relationships can adaptively encode spatiotemporal patterns without frequency-domain constraints. The theoretical contribution—approximation error bounds as functions of domain boundary complexity κ—provides principled guidance for practitioners choosing between architectures rather than relying on empirical intuition.

The ablation studies reveal a critical practical insight: physics regularization terms create an inductive bias tradeoff. While these priors improve diffusion-dominated problems, they degrade performance on chaotic and recirculating-flow regimes where physics assumptions misalign with actual dynamics. This prior misspecification boundary directly impacts how scientists should configure these models for different physical regimes. For the broader AI field, the work demonstrates that architectural innovations remain crucial even within the transformer paradigm, with domain-specific constraints driving design choices rather than one-size-fits-all solutions.

• →MSAT achieves 3.7x better accuracy than Fourier Neural Operators on complex geometry PDE problems while running 3,500x faster than Mamba-NO.
• →Attention mechanisms outperform Fourier-based methods on irregular domains due to adaptive learned relationships rather than frequency-domain constraints.
• →Physics regularization creates a precision tradeoff: beneficial for diffusion problems but harmful for chaotic flows, defining a clear prior-misspecification boundary.
• →Theoretical approximation bounds parameterized by domain boundary complexity provide principled architecture selection rules beyond empirical benchmarking.
• →The research demonstrates architectural innovations within transformers remain critical for scientific computing rather than treating transformers as a uniform solution class.