How Context Shapes Truth: Geometric Transformations of Statement-level Truth Representations in LLMs

Researchers demonstrate that Large Language Models encode truth as geometric vectors in their activation space, and these vectors undergo predictable transformations when contextual information is introduced. The study reveals that larger models rely on directional changes to distinguish relevant context while smaller models use magnitude shifts, with conflicting context producing larger geometric shifts than aligned context.

This research advances our understanding of how LLMs internally represent and process truth claims, a critical area for AI safety and interpretability. By examining truth vectors across multiple models and datasets, the authors establish that context doesn't randomly perturb these representations but follows consistent geometric patterns. Early layers show orthogonal truth vectors suggesting independence from context, while middle layers demonstrate convergence, indicating where contextual integration occurs.

The finding that larger models primarily distinguish relevant from irrelevant context through directional rotation rather than magnitude scaling has significant implications for model robustness and hallucination mitigation. This suggests different computational strategies emerge at scale. The observation that conflicting context produces larger geometric changes than parametrically aligned context reveals how models manage competing information sources—a practical concern for real-world deployment where user inputs frequently contradict training data.

For AI developers and safety researchers, this work provides actionable geometric interpretability tools. Understanding how truth representations transform enables better detection of when models might prioritize context over accurate knowledge. The methodology could inform techniques for controlling model behavior through activation manipulation, potentially reducing false outputs without full retraining.

Looking ahead, this geometric characterization framework may enable more targeted interventions in LLM behavior. Researchers can now target specific layers and directions where context integration occurs, potentially creating more reliable safeguards against context-induced hallucinations and improving performance on tasks requiring balanced knowledge integration.

• →Truth vectors in LLMs undergo predictable geometric transformations that vary systematically across network layers
• →Larger models distinguish relevant context primarily through directional rotation, while smaller models rely on magnitude changes
• →Adding context generally amplifies the separation between true and false representations in activation space
• →Conflicting context produces larger geometric changes than context aligned with parametric knowledge
• →This geometric framework enables more targeted interpretability approaches for improving LLM reliability