Researchers introduce a geometric decomposition framework to understand how prompting reshapes internal representations in large language models and vision-language models without weight updates. Testing across multiple models and datasets reveals that prompts consistently reorganize representations toward task structures, with cross-dimensional linear mixing (affine transformations) emerging as a key mechanism for prompt-driven behavior.
This research addresses a fundamental question in AI interpretability: how do prompts actually steer model behavior at the representational level? Rather than treating prompting as a black box, the authors develop a systematic framework that maps how instruction changes transform the geometric structure of internal activations. This matters because understanding these mechanisms is essential for building more reliable, controllable AI systems.
The nested decomposition approach is methodologically sophisticated, starting with simple geometric transformations (translation, rigid scaling) and progressing to complex nonlinear remapping. By causally intervening on individual layers and measuring whether mapped representations recover target task geometry, the researchers move beyond correlation to establish mechanism. The finding that affine transformations—which involve cross-dimensional linear mixing—are necessary to fully recover task-specific geometry is particularly significant, suggesting prompts operate through systematic reorganization of feature interactions rather than simple shifts or rotations.
For AI development, this work provides actionable insights into prompt effectiveness and generalization. Understanding that prompts leverage linear mixing mechanisms could inform better prompt engineering strategies and improve transfer across domains. The layer-by-layer routing patterns identified suggest models have learned specialized strategies for different tasks, which has implications for multitask learning and instruction following.
The research establishes interpretability foundations that will likely influence how practitioners design prompts and evaluate model alignment. Future work testing these mechanisms in larger models and adversarial settings could reveal prompt robustness limitations and inform safety research. This represents progress toward AI systems whose behavior can be understood and predicted through geometric principles.
- →Prompts reshape model representations through interpretable geometric transformations, with affine transformation (linear mixing) being key to recovering task-specific geometry
- →Cross-validated analysis across six datasets and multiple architectures shows consistency in how models route task-relevant structure through layers
- →Simple transformations like translation and rigid scaling explain significant variance, suggesting prompt efficiency operates through elegant geometric principles
- →The framework enables causal testing of representational changes by layer, revealing model-specific routing strategies for different task types
- →Understanding prompt mechanisms as geometric transformations provides foundations for improving prompt engineering and AI interpretability practices