Boosting In-Context Learning in LLMs Through the Lens of Classical Supervised Learning

Researchers propose Supervised Calibration (SC), a new framework to improve In-Context Learning performance in Large Language Models by addressing systematic biases through optimal affine transformations in logit space. The method achieves state-of-the-art results across multiple LLMs including Mistral-7B, Llama-2-7B, and Qwen2-7B in few-shot learning scenarios.

• →Current calibration techniques for LLMs only shift decision boundaries without changing their orientation, proving inadequate for severely misaligned models.
• →Supervised Calibration (SC) learns optimal per-class transformations of LLM predictive probabilities without requiring external data beyond context.
• →SC subsumes many existing calibration methods as special cases while enabling complete reversal of decision boundary orientation.
• →The framework integrates context-invariance and directional trust-region regularizers to tackle instability and control calibration degree.
• →SC delivers state-of-the-art performance across nine datasets in 4-shot, 8-shot, and 16-shot settings for three major 7B parameter models.