Superposition Is Not Necessary: A Mechanistic Interpretability Analysis of Transformer Representations for Time Series Forecasting

Researchers applied mechanistic interpretability tools to analyze how transformer models process time series data, discovering that these models don't rely on superposition—a complex representational technique crucial to their NLP success. The findings explain why simpler linear models remain competitive for forecasting and suggest transformers may be overengineered for standard time series benchmarks.

This mechanistic interpretability study addresses a longstanding puzzle in time series modeling: why simple linear models like DLinear consistently match or approach the performance of sophisticated transformer architectures. Researchers probed the internal representations of PatchTST using sparse autoencoders, systematically expanding dictionary sizes to detect whether the model compresses multiple concepts into single neurons—a hallmark of superposition observed in language models.

The analysis reveals transformers achieve competitive forecasting performance through sparse, straightforward representations that remain stable under aggressive dictionary expansion. Causal interventions on dominant latent features produced minimal forecast disruption, indicating the model's success doesn't depend on intricate feature interactions. This contrasts sharply with transformer behavior in NLP, where superposition enables handling of compositional language tasks.

These findings carry significant implications for AI infrastructure and model selection in time series applications. Organizations may be deploying unnecessarily complex architectures when simpler approaches suffice, creating wasteful computational overhead. The research suggests standard forecasting benchmarks lack the compositional richness that justifies transformer complexity, potentially explaining why domain-specific models haven't achieved expected performance gains despite architectural sophistication.

The work highlights a critical gap between transformer capabilities and practical requirements for forecasting tasks. This mechanistic understanding enables more informed architecture choices, guiding developers toward efficiency rather than following architectural trends from language modeling. Future research should explore whether specialized forecasting tasks with higher compositional demands would activate superposition mechanisms and justify added complexity.

• →Transformers for time series forecasting rely on sparse, simple representations rather than superposition—the complex encoding mechanism crucial to their NLP success
• →Single-layer, narrow-dimensional transformers match deeper configurations across standard benchmarks, questioning the necessity of architectural depth
• →Dictionary expansion to 4x native dimensionality produces negligible performance changes with large portions remaining inactive, indicating representational inefficiency
• →Standard time series forecasting benchmarks may lack compositional complexity required to justify transformer adoption over linear models
• →Mechanistic interpretability reveals simple linear models' persistent competitiveness stems from forecasting tasks' lower representational demands