AIBearisharXiv – CS AI · 7h ago7/10
🧠
Categorical Prior Lock-in: Why In-Context Learning Fails for Structured Data
Researchers identify a fundamental limitation in large language models' ability to adapt to structured data through in-context learning, discovering that LLMs fail to update their categorical token distributions learned during pre-training even with additional examples. While parameter-efficient fine-tuning overcomes this constraint, it introduces memorization risks and potential instability in structured output generation.