#annotation-quality News & Analysis

4 articles tagged with #annotation-quality. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

4 articles

AIBearisharXiv – CS AI · Jun 87/10

🧠

Does Topic Sentiment Cause Perceived Ideology? Comparing Human and LLM Annotations in Political News Articles

A research study compares how human annotators and large language models (GPT-4o-mini, Llama-3.3-70B) assign political ideology labels to news articles, finding that fine-tuned GPT-4o-mini models develop spurious correlations between sentiment and ideology that don't exist in human judgment. This reveals a critical vulnerability in using LLM annotations as training data for downstream tasks.

🧠 GPT-4🧠 Llama

AINeutralarXiv – CS AI · Jun 236/10

🧠

LLM-Based Multi-Reference Evaluation for Efficient and Robust Assessment of Phrase Break Annotations

Researchers propose LLM-Based Multi-Reference Evaluation (LMRE), a new method for assessing phrase break annotations in speech that acknowledges multiple valid phrasings rather than assuming a single correct interpretation. Tested on 1,356 Korean annotations, LMRE demonstrates stronger alignment with human judgment than traditional single-reference approaches, suggesting large language models can effectively evaluate prosodic speech characteristics at scale.

AINeutralarXiv – CS AI · Jun 96/10

🧠

Deep Active Re-Labeling: Toward Noise-Resilient Annotation Efficiency

Researchers propose Deep Active Re-Labeling (DARL), a framework addressing human annotation errors in deep active learning by allocating budget to re-annotate potentially mislabeled data. The method uses noise detection strategies to identify suspect instances, improving data quality and model performance under annotation noise.

AINeutralarXiv – CS AI · Jun 26/10

🧠

Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025

A comprehensive audit of 1,603 NLP papers from 2018-2025 reveals that while researchers increasingly report operational annotation details like recruitment and expertise, critical information for assessing data validity—such as annotator training, language proficiency, compensation, and inter-annotator agreement—remains frequently omitted. The study establishes a scalable framework and reporting taxonomy to improve reproducibility and reliability in NLP research.