#research-methods News & Analysis

3 articles tagged with #research-methods. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

3 articles

AINeutralarXiv – CS AI · 6d ago6/10

🧠

Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025

A comprehensive audit of 1,603 NLP papers from 2018-2025 reveals that while researchers increasingly report operational annotation details like recruitment and expertise, critical information for assessing data validity—such as annotator training, language proficiency, compensation, and inter-annotator agreement—remains frequently omitted. The study establishes a scalable framework and reporting taxonomy to improve reproducibility and reliability in NLP research.

AINeutralarXiv – CS AI · Apr 206/10

🧠

Distribution Shift Alignment Helps LLMs Simulate Survey Response Distributions

Researchers introduced Distribution Shift Alignment (DSA), a novel fine-tuning method that enables large language models to more accurately simulate human survey responses by learning distribution patterns rather than memorizing training data. DSA outperforms existing methods across five public datasets and reduces required real-world data by 53-69%, offering significant cost savings for large-scale survey research.

AINeutralarXiv – CS AI · Mar 34/103

🧠

When Numbers Tell Half the Story: Human-Metric Alignment in Topic Model Evaluation

Researchers introduce Topic Word Mixing (TWM), a new human evaluation method for assessing topic models in specialized domains. The study reveals misalignment between automated metrics and human judgment, particularly in domain-specific corpora like philosophy of science publications.