Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs

Researchers discovered that Large Language Models become increasingly sparse in their internal representations when handling more difficult or out-of-distribution tasks. This sparsity mechanism appears to be an adaptive response that helps stabilize reasoning under challenging conditions, leading to the development of a new learning strategy called Sparsity-Guided Curriculum In-Context Learning (SG-ICL).