Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation

Researchers conducted a large-scale analysis of human evaluation protocols across 284 *CL conference papers (2023-2025), discovering widespread under-reporting of critical study design details that undermine reproducibility. The findings reveal that transparency gaps in how text generation quality is assessed create ambiguity about measurement methodology, evaluator credentials, and result interpretation, prompting actionable recommendations for improved reporting standards.

This research addresses a foundational crisis in AI evaluation methodology that directly impacts the reliability of published findings in natural language processing. The study's scope—examining 284 papers manually plus 1,800+ with LLM assistance—provides robust evidence that the field lacks standardized transparency in human evaluation, a practice considered the gold standard for assessing text generation quality. The identified 20 reportable criteria span critical dimensions: evaluator expertise, annotation guidelines, inter-annotator agreement metrics, and conflict resolution procedures. Current under-reporting creates a reproducibility bottleneck where downstream researchers cannot verify findings or build comparable studies.

The broader context reflects AI's rapid advancement outpacing governance infrastructure. As large language models proliferate, evaluation rigor becomes increasingly important for distinguishing genuine improvements from statistical artifacts or methodological artifacts. Papers lacking transparent protocols may overstate model capabilities or mask systematic biases in assessment. This pattern parallels reproducibility crises in other scientific domains where publication pressure incentivizes incomplete methodology reporting.

For practitioners and organizations deploying text generation systems, this analysis signals caution: published benchmarks may not translate reliably to production environments. The transparency gaps make it harder to assess whether models genuinely improve language understanding or merely exploit evaluation loopholes. For the research community, the actionable recommendations—detailed annotation schemas, evaluator qualification disclosure, and standardized reporting templates—establish clearer pathways toward reproducible science.

Moving forward, conference review processes should enforce the identified reporting criteria as publication prerequisites. Adoption of these standards would strengthen the evidence base for AI capabilities while enabling more confident technology adoption across industries.

• →284 reviewed papers show systematic under-reporting of human evaluation protocols, compromising reproducibility in text generation assessment
• →20 reportable criteria framework identifies critical missing details: evaluator expertise, annotation procedures, and inter-annotator agreement metrics
• →Transparency gaps create ambiguity about what was actually measured, potentially inflating model performance claims through methodological artifacts
• →Lack of standardized reporting protocols hinders downstream researchers from verifying findings or building comparable studies
• →Actionable recommendations include mandatory disclosure of annotation schemas, evaluator qualifications, and conflict resolution procedures in publications