DiZiNER: Disagreement-guided Instruction Refinement via Pilot Annotation Simulation for Zero-shot Named Entity Recognition

Researchers introduce DiZiNER, a framework that improves zero-shot named entity recognition by simulating human annotation disagreement processes using multiple LLMs. The approach achieves state-of-the-art results on 14 of 18 benchmarks, closing the performance gap between zero-shot and supervised systems by over 11 percentage points.

DiZiNER addresses a fundamental limitation in how large language models perform information extraction tasks without task-specific training data. The framework's innovation lies in treating multiple heterogeneous LLMs as a diverse annotation team, where disagreements between models signal areas requiring instruction refinement. This mirrors proven methodologies from human annotation workflows where pilot studies identify and resolve systematic inconsistencies.

The research builds on growing recognition that instruction quality directly impacts zero-shot performance in LLMs. Prior work showed instruction fine-tuning improves generative outputs, yet zero-shot NER remained substantially weaker than supervised approaches. DiZiNER's contribution demonstrates that the gap stems partly from suboptimal instructions rather than fundamental model limitations. The supervisor model analyzes disagreement patterns to iteratively improve prompts, creating a feedback loop that enhances task clarity.

For the AI development community, these results validate that prompt engineering at scale—using multiple models to identify failure modes—outperforms single-model optimization. The finding that improvements exceed what the supervisor model (GPT-4 mini) alone achieves proves the value derives from disagreement-guided refinement rather than raw model capacity. This has implications for cost-effective AI deployment, suggesting engineers can extract superior performance from accessible models through better instruction design.

The 8.0 F1 improvement and 11-point gap reduction across 18 benchmarks demonstrate broad applicability. Practitioners deploying zero-shot NER systems for information extraction could adopt similar disagreement-based refinement strategies. Future work likely explores whether this approach generalizes to other IE tasks and whether human annotators could augment the disagreement analysis process.

• →DiZiNER achieves zero-shot SOTA on 14 of 18 NER benchmarks by using disagreement between multiple LLMs to refine task instructions.
• →The framework reduces the performance gap between zero-shot and supervised systems by over 11 F1 points, addressing a major limitation in information extraction.
• →Improvements stem from instruction quality rather than model capacity, as performance exceeds the individual supervisor model's baseline.
• →Multi-model disagreement analysis correlates strongly with NER performance, validating the approach's theoretical foundation.
• →The methodology offers practical applications for cost-effective deployment of zero-shot NER systems in production environments.