On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance

Researchers demonstrate that Large Language Models exhibit significant limitations in zero-shot annotation tasks, with only 34.8% of initial errors correctable through prompting. The study reveals that model-internalized priors and concept definitions strongly influence LLM performance more than text-level memorization, highlighting fundamental constraints in LLM adaptability for reliable AI-as-a-judge applications.

This research exposes a critical vulnerability in the widespread deployment of LLMs for annotation and evaluation tasks. The finding that nearly two-thirds of zero-shot errors resist correction through additional prompting contradicts assumptions about LLM flexibility and raises serious questions about reliability in production systems. When LLMs encounter high-confidence errors, they demonstrate remarkable stubbornness—additional context fails to dislodge entrenched predictions, suggesting these errors stem from deep architectural biases rather than insufficient information.

The study's introduction of Definition-Specific Familiarity (DSF) as a performance metric represents a significant conceptual advance. By measuring alignment between a model's learned concept and a task's formal definition, DSF outperforms traditional memorization metrics like ROUGE-L and BERTScore in predicting performance. This finding fundamentally reframes the problem: LLM failures aren't primarily about training data memorization but rather about foundational concept misalignment built into model weights during pretraining.

The troubling discovery that LLMs follow misaligned task definitions while maintaining unchanged confidence levels exposes a particularly dangerous failure mode. Systems appear equally certain whether following correct or incorrect instructions, eliminating confidence as a reliability signal. For organizations deploying LLMs in content moderation, legal review, or scientific annotation, these constraints demand immediate reassessment of trust assumptions.

Moving forward, practitioners cannot rely on prompt engineering to reliably correct model behavior. Instead, development effort must focus on either identifying tasks where model-internalized priors naturally align with definitions, or implementing human-in-the-loop validation for high-stakes decisions. This research suggests the frontier of LLM improvement lies not in better prompting but in addressing fundamental architectural constraints.

• →Only 34.8% of LLM zero-shot annotation errors can be corrected through additional prompting, revealing fundamental adaptability limits
• →High-confidence errors prove resistant to correction, making model confidence unreliable as a quality signal
• →Definition-Specific Familiarity predicts performance better than traditional memorization metrics, indicating concept alignment matters more than training data recall
• →LLMs follow misaligned task definitions while maintaining identical confidence levels, creating dangerous failure modes in critical applications
• →Prompt-based correction strategies have inherent limits; architectural constraints may require alternative validation approaches for reliable annotations