Understanding In-Context Learning Beyond Transformers: An Investigation of State Space and Hybrid Architectures

Researchers conducted an in-depth analysis of in-context learning capabilities across different AI architectures including transformers, state-space models, and hybrid systems. The study reveals that while these models perform similarly on tasks, their internal mechanisms differ significantly, with function vectors playing key roles in self-attention and Mamba layers.

• →Different AI architectures achieve similar task performance but use fundamentally different internal mechanisms for in-context learning.
• →Function vectors responsible for in-context learning are primarily located in self-attention and Mamba layers.
• →Mamba2 architecture appears to use a different mechanism than function vectors for performing in-context learning.
• →Function vectors are more critical for parametric knowledge retrieval tasks than contextual knowledge understanding.
• →The research emphasizes the importance of combining behavioral and mechanistic analysis methods to understand AI capabilities.