Categorical Prior Lock-in: Why In-Context Learning Fails for Structured Data

Researchers identify a fundamental limitation in large language models' ability to adapt to structured data through in-context learning, discovering that LLMs fail to update their categorical token distributions learned during pre-training even with additional examples. While parameter-efficient fine-tuning overcomes this constraint, it introduces memorization risks and potential instability in structured output generation.

The research exposes a critical architectural weakness in how LLMs handle structured data generation. When presented with new categorical distributions through in-context examples, models exhibit 'categorical prior lock-in'—they retain their original pre-trained distributions and cannot flexibly adapt to rare classes or novel categories in tabular data. This represents a fundamental mismatch between how LLMs learn and what structured data tasks require.

This discovery emerges as LLMs increasingly serve as conditional generators across enterprises managing complex tabular datasets. The findings challenge the prevailing assumption that in-context learning provides a flexible, parameter-efficient adaptation mechanism across all data modalities. While ICL successfully improves numerical accuracy, the categorical ceiling suggests models are applying statistical priors rather than genuinely updating their understanding of data distributions.

The practical implications are substantial for developers deploying LLMs in data generation and augmentation pipelines. Organizations relying on in-context adaptation for categorical features will face systematic failures on underrepresented classes, degrading data quality and downstream analytics. The proposed alternative—LoRA fine-tuning—trades this adaptability problem for privacy vulnerabilities and occasional output instability, forcing practitioners to choose between flexibility and security.

Looking forward, this research highlights the need for hybrid approaches that separate categorical and numerical generation pathways, or architectural modifications enabling genuine distribution updates. The work also suggests that specialized tabular models may retain advantages over generalist LLMs for structured data tasks, challenging the narrative of LLM universality.

• →LLMs cannot effectively update categorical token distributions through in-context learning, causing systematic failure on rare classes.
• →Numerical features show continuous improvement with additional examples while categorical features plateau sharply regardless of prompt engineering.
• →Parameter-efficient fine-tuning (LoRA) solves adaptability but introduces measurable memorization risks and occasional output instability.
• →The categorical prior lock-in failure mode reveals a fundamental gap between LLM learning mechanisms and structured data generation requirements.
• →Organizations may need hybrid architectures or specialized models for high-cardinality tabular data rather than relying on generalist LLMs.