Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias

Researchers discover that the 'Lost in the Middle' phenomenon in transformer models - where AI performs poorly on middle context but well on beginning and end content - is an inherent architectural property present even before training begins. The U-shaped performance bias stems from the mathematical structure of causal decoders with residual connections, creating a 'factorial dead zone' in middle positions.

• →The U-shaped performance bias exists at model initialization before any training or positional encoding takes effect.
• →Causal masking creates strong gradient influence at the prompt start while residual connections anchor the final token position.
• →Middle context positions form a factorial dead zone of order O(1/(H-1)!) where H is network depth, making retrieval structurally difficult.
• →Standard pretraining does not overcome this architectural baseline, as confirmed in untrained Qwen2 and GPT-2 models.
• →The research establishes the mathematical foundation for future interventions to address this inherent transformer limitation.