How Transformers Reject Wrong Answers: Rotational Dynamics of Factual Constraint Processing

Researchers discovered that transformer language models process factual information through rotational dynamics rather than magnitude changes, actively suppressing incorrect answers instead of passively failing. This geometric pattern only emerges in models above 1.6B parameters, suggesting a phase transition in factual processing capabilities.

• →Language models distinguish correct from incorrect answers through rotational changes in vector direction, not magnitude scaling.
• →Models actively suppress correct answers when processing incorrect continuations rather than passively failing.
• →Factual constraint processing capabilities emerge only above 1.6B parameters, indicating a critical threshold.
• →The geometric character of factual processing is invisible to traditional single-layer probing methods.
• →Internal representations diverge across network depth through angular separation while maintaining similar magnitudes.