77 articles tagged with #in-context-learning. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers demonstrate that Mamba, a state space model alternative to transformers, efficiently learns optimal statistical estimators for Markov chains through in-context learning. The study reveals that single-layer Mamba discovers the Laplacian smoothing estimator—which is both Bayes and minimax optimal—and theoretically explains this capability through convolution-based representation.
Researchers analyzed how large language models interpret mixed compliance demonstrations—combining benign and harmful requests with helpful responses—revealing that demonstration composition critically affects model behavior. The study shows that benign demonstrations can either reduce or increase harmful compliance depending on the model, with preference optimization during training and demonstration ordering playing crucial roles in preventing jailbreaks.
Researchers propose that reinforcement learning foundation models should be developed using synthetic MDPs (Markov Decision Processes) as training data, similar to how TabPFN uses synthetic data for tabular prediction. A Graph Attention Network trained entirely on synthetic MDPs demonstrates strong performance on both online and offline RL benchmarks without task-specific tuning, suggesting this approach is viable.
Researchers introduce TASM (Task-Aware Structured Memory), a training-free framework that optimizes how multi-modal large language models compress and retrieve information during in-context learning. The method addresses critical scalability limitations by using task-aware compression, structure-preserving token merging, and dynamic memory hierarchies to maintain performance while reducing computational costs.
Researchers identify a fundamental limitation in large language models' ability to adapt to structured data through in-context learning, discovering that LLMs fail to update their categorical token distributions learned during pre-training even with additional examples. While parameter-efficient fine-tuning overcomes this constraint, it introduces memorization risks and potential instability in structured output generation.
Researchers have identified critical performance gaps in open-source Relational Foundation Models (RFMs) compared to commercial alternatives by analyzing the Relational Transformer architecture. Their findings—that sparse label coverage and insufficient real-world training data limit current models—led to OpenRFM, which achieves 30% performance improvements and outperforms the commercial KumoRFMv1 baseline.
Researchers introduce ICICLE, a generative retrieval framework that addresses the inefficiency of traditional corpus expansion by treating new documents as in-context evidence rather than requiring model retraining. The approach uses a copy-based routing mechanism to distinguish between parametric memory and context-provided document associations, achieving better scalability without catastrophic forgetting.
Researchers demonstrate that integrating reinforcement learning objectives into offline in-context RL frameworks significantly outperforms supervised learning approaches like Algorithm Distillation, achieving ~30% performance improvements across diverse environments and doubling performance in complex settings. The findings validate that aligning ICRL training with RL reward-maximization goals, particularly through conservative value learning, produces more effective agents.
Researchers demonstrate that transformer models equipped with continuous latent context tokens can efficiently implement online learning algorithms without parameter updates. A small GPT-2-style model trained with this approach outperforms much larger language models on synthetic online prediction tasks, suggesting a promising architectural direction for adaptive AI systems.
Researchers propose a framework for optimizing data selection in large language model instruction tuning by learning task-specific and model-specific weights for multiple quality indicators. Using efficient in-context learning signals on small validation sets, the method achieves comparable performance to full-dataset training with only 30% of samples, revealing important trade-offs between semantic diversity and logical complexity.
Researchers demonstrate that large language models suffer from 'in-context fixation,' where homogeneous demonstration labels—even semantically valid ones—cause classification accuracy to collapse below 12%. The models treat label-slot tokens as an exhaustive vocabulary set rather than learning from semantic meaning, revealing that in-context learning operates as constrained vocabulary retrieval rather than genuine concept learning.
A new research study reveals that large language models struggle to effectively use representations they learn from in-context information, even though they can encode this information internally. The findings suggest current LLMs have fundamental limitations in adapting to novel contexts, affecting their ability to generalize learned patterns to downstream tasks.
Research shows that in-context examples in large language models suppress recall of scientific knowledge, causing models to shift from knowledge-driven reasoning to empirical pattern fitting even when examples are generated from the same formulas they should reinforce. This finding across 60 tasks and four models suggests practitioners deploying LLMs for scientific work should be cautious about using examples, as they may undermine rather than support domain expertise.
Researchers published a comprehensive technical survey on Large Language Model augmentation strategies, examining methods from in-context learning to advanced Retrieval-Augmented Generation techniques. The study provides a unified framework for understanding how structured context at inference time can overcome LLMs' limitations of static knowledge and finite context windows.
Researchers have developed techniques to mitigate many-shot jailbreaking (MSJ) attacks on large language models, where attackers use numerous examples to override safety training. Combined fine-tuning and input sanitization approaches significantly reduce MSJ effectiveness while maintaining normal model performance.
Researchers developed Localized In-Context Learning (L-ICL), a technique that significantly improves large language model performance on symbolic planning tasks by targeting specific constraint violations with minimal corrections. The method achieves 89% valid plan generation compared to 59% for best baselines, representing a major advancement in LLM reasoning capabilities.
Researchers studied how large language models generalize to new tasks through "off-by-one addition" experiments, discovering a "function induction" mechanism that operates at higher abstraction levels than previously known induction heads. The study reveals that multiple attention heads work in parallel to enable task-level generalization, with this mechanism being reusable across various synthetic and algorithmic tasks.
Researchers propose Supervised Calibration (SC), a new framework to improve In-Context Learning performance in Large Language Models by addressing systematic biases through optimal affine transformations in logit space. The method achieves state-of-the-art results across multiple LLMs including Mistral-7B, Llama-2-7B, and Qwen2-7B in few-shot learning scenarios.
Researchers introduce RDB-PFN, the first relational foundation model for databases trained entirely on synthetic data to overcome privacy and scarcity issues with real relational databases. The model uses a Relational Prior Generator to create over 2 million synthetic tasks and demonstrates strong few-shot performance on 19 real-world relational prediction tasks through in-context learning.
Researchers discovered that Large Language Models become increasingly sparse in their internal representations when handling more difficult or out-of-distribution tasks. This sparsity mechanism appears to be an adaptive response that helps stabilize reasoning under challenging conditions, leading to the development of a new learning strategy called Sparsity-Guided Curriculum In-Context Learning (SG-ICL).
Researchers propose Many-Shot In-Context Fine-tuning (ManyICL), a novel approach that significantly improves large language model performance by treating multiple in-context examples as supervised training targets rather than just prompts. The method narrows the performance gap between in-context learning and dedicated fine-tuning while reducing catastrophic forgetting issues.
Research compares Transformers, State Space Models (SSMs), and hybrid architectures for in-context retrieval tasks, finding hybrid models excel at information-dense retrieval while Transformers remain superior for position-based tasks. SSM-based models develop unique locality-aware embeddings that create interpretable positional structures, explaining their specific strengths and limitations.
Researchers introduce Spectrum Tuning, a new post-training method that improves AI language models' ability to generate diverse outputs and follow in-context steering instructions. The technique addresses limitations in current post-training approaches that reduce models' distributional coverage and flexibility when tasks require multiple valid answers rather than single correct responses.
Researchers demonstrate that large language models can match or exceed fine-tuned BERT performance on Named Entity Recognition tasks when provided with hundreds of in-context examples rather than just a few. The study shows many-shot in-context learning can also serve as a data annotation framework, generating high-quality training data that improves low-resource NER by ~10% F1 when used to fine-tune supervised models.
Researchers introduce T2J, a benchmark dataset of PyTorch-to-JAX translation bugs paired with developer fixes, addressing the challenge of translating deep-learning code between frameworks. By training LLMs on this curated bug-fix data through in-context learning, they achieve up to 20% improvement in translation accuracy, demonstrating that domain-specific bug datasets can significantly enhance code generation reliability.
