Diagnosing Retrieval Bias Under Multiple In-Context Knowledge Updates in Large Language Models

Researchers identify a significant bias in Large Language Models when processing multiple updates to the same factual information within context. The study reveals that LLMs struggle to accurately retrieve the most recent version of updated facts, with performance degrading as the number of updates increases, similar to memory interference patterns observed in cognitive psychology.

• →LLMs exhibit retrieval bias when the same fact is updated multiple times in context, performing worse on latest information while maintaining accuracy on earliest states.
• →The bias intensifies as the number of knowledge updates increases, creating a persistent challenge for long-context applications.
• →Diagnostic analysis shows attention patterns and hidden states become less discriminative during errors, making it harder to identify current information.
• →Current heuristic intervention strategies provide only modest improvements and fail to eliminate the underlying bias.
• →The research introduces a Dynamic Knowledge Instance framework for systematically evaluating multi-update scenarios in LLMs.