A Geometric Account of Activation Steering through Angle-Norm Decomposition

Researchers present a geometric framework for understanding activation steering in language models by decomposing interventions into angular and radial components. The study finds that while concepts are primarily encoded in angular structure, the hidden-state norm remains important for steering stability and effectiveness, suggesting that steering methods should be parameterized separately for these two geometric effects rather than as a single additive coefficient.

This research addresses a fundamental question in mechanistic interpretability: how do we effectively control language model behavior through activation steering? The work bridges two competing approaches—linear additive steering and spherical steering paradigms—by providing empirical evidence that both angular and radial components matter, despite prior assumptions that norm carries no concept-relevant information.

The significance lies in the geometric insight that steering interventions couple two distinct effects. By conducting controlled experiments across seven language models, the researchers demonstrate that concepts cluster along specific angular directions in representation space, validating the spherical steering motivation. However, the discovery that norm influences downstream stability and effects contradicts the assumption that it can be ignored. This finding explains previously puzzling observations where interventions with similar concept-level effects produce different behavioral outcomes.

For the AI research community, this work has practical implications for model safety and alignment. Better understanding of steering mechanisms enables more precise control over model outputs, which is crucial for preventing unwanted behaviors. Developers working on activation steering implementations can now design more interpretable and stable interventions by explicitly parameterizing angular and radial components separately.

The research suggests that future steering methods should treat these geometric dimensions independently rather than entangling them in a single coefficient. This could lead to more efficient interventions, better generalization across contexts, and improved interpretability of what steering actually accomplishes. As language models grow larger and more capable, precise steering mechanisms become increasingly important for safe deployment.

• →Activation steering effects decompose into angular alignment and norm changes, two geometrically distinct mechanisms that interact in complex ways.
• →Concepts are encoded primarily in angular structure of hidden states, supporting spherical steering approaches over purely additive methods.
• →Hidden-state norm remains important for steering stability and downstream effects despite not directly encoding concept information.
• →Different steering interventions can produce similar concept-level effects while behaving differently due to varying norm impacts.
• →Parameterizing steering with separate angular and radial components improves interpretability and control over model behavior.