Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs

Researchers discovered that Large Language Models become increasingly sparse in their internal representations when handling more difficult or out-of-distribution tasks. This sparsity mechanism appears to be an adaptive response that helps stabilize reasoning under challenging conditions, leading to the development of a new learning strategy called Sparsity-Guided Curriculum In-Context Learning (SG-ICL).

• →LLMs exhibit increasing sparsity in their last hidden states as task difficulty and out-of-distribution shift increases.
• →This sparsity-difficulty relationship is consistent across diverse models and domains, suggesting it's a fundamental adaptive mechanism.
• →The sparsity response helps LLMs concentrate computation into specialized subspaces when encountering unfamiliar or complex inputs.
• →Researchers developed SG-ICL, a strategy that uses representation sparsity to schedule few-shot demonstrations for performance improvements.
• →The study provides new mechanistic insights into how LLMs internally process and adapt to challenging inputs.