OrthoFormer: Instrumental Variable Estimation in Transformer Hidden States via Neural Control Functions

Researchers propose OrthoFormer, a new Transformer architecture that addresses causal learning limitations by embedding instrumental variable estimation directly into neural networks. The framework aims to distinguish between spurious correlations and true causal mechanisms, potentially improving AI model robustness and reliability under distribution shifts.

• →OrthoFormer integrates causal inference principles directly into Transformer architectures to overcome correlational learning limitations.
• →The framework uses four theoretical pillars including structural directionality and representation orthogonality to separate causal flows from background noise.
• →Theoretical analysis proves OrthoFormer achieves lower bias than ordinary least squares for valid instrument lags.
• →The research identifies a 'neural forbidden regression' phenomenon where improved prediction can destroy causal validity.
• →This represents a potential paradigm shift from correlational to causal sequence modeling in AI systems.