Scaling Performance and Low-Resource Annotation with Many-Shot In-Context Learning for Named Entity Recognition

Researchers demonstrate that large language models can match or exceed fine-tuned BERT performance on Named Entity Recognition tasks when provided with hundreds of in-context examples rather than just a few. The study shows many-shot in-context learning can also serve as a data annotation framework, generating high-quality training data that improves low-resource NER by ~10% F1 when used to fine-tune supervised models.

This research addresses a fundamental limitation in applying large language models to structured prediction tasks like Named Entity Recognition, where traditional supervised approaches have historically maintained an edge. The gap between LLM capabilities and fine-tuned models in NER stems from the complexity of precise token-level labeling and the need for tight integration with task-specific architectures. The study's key contribution—demonstrating that scaling in-context examples from dozens to hundreds significantly narrows this performance gap—represents a meaningful shift in how practitioners might approach NER workflows.

The findings emerge from a landscape where few-shot learning has become standard practice for rapid prototyping, yet hasn't consistently matched supervised baselines for sequence labeling. By systematically evaluating hundreds of demonstrations across multiple domains, researchers identified a scaling relationship that prior work overlooked. This reflects broader trends in LLM capabilities where additional context and examples often unlock latent performance improvements.

For practitioners and organizations, the implications are pragmatic. Many-shot in-context learning reduces the need for extensive manual annotation cycles while generating synthetic training data that maintains competitive quality. The ~10% F1 improvement on low-resource tasks demonstrates concrete value beyond convenience—it translates to measurable accuracy gains where labeled data remains scarce. This hybrid approach combining LLMs for initial annotation with supervised fine-tuning creates an efficient annotation pipeline applicable across domains.

The research leaves open questions about computational costs, latency tradeoffs during inference, and performance on highly domain-specific entity types. Future work should examine whether these gains persist across more specialized verticals and how many-shot methods scale to multilingual NER scenarios.

• →Scaling in-context examples to hundreds enables LLMs to match or exceed fine-tuned BERT performance on NER tasks
• →Many-shot ICL can generate high-quality labeled data from ~100 human examples for downstream fine-tuning
• →Hybrid approaches combining LLM annotation with supervised models achieve ~10% absolute F1 improvement in low-resource settings
• →In-context learning eliminates training overhead while maintaining competitive accuracy for structured prediction tasks
• →The method reduces annotation burden while providing an efficient framework for bootstrapping labeled datasets