AINeutralarXiv – CS AI · 7h ago6/10
🧠
Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025
A comprehensive audit of 1,603 NLP papers from 2018-2025 reveals that while researchers increasingly report operational annotation details like recruitment and expertise, critical information for assessing data validity—such as annotator training, language proficiency, compensation, and inter-annotator agreement—remains frequently omitted. The study establishes a scalable framework and reporting taxonomy to improve reproducibility and reliability in NLP research.