Before the Labels: How Dataset Construction Shapes Suicidality Detection in Clinical Text

Researchers demonstrate that clinical NLP datasets for suicidality detection, particularly the ScAN dataset built on MIMIC-III notes, embed specific operational choices that obscure how labels are constructed rather than representing objective ground truth. The study reveals that dataset design decisions—including single annotators, ICD-based cohort selection, and hospital-stay aggregation—shape what suicidality means in algorithmic systems, highlighting critical gaps between documented clinical judgments and actual suicidal intent.

This research addresses a fundamental problem in clinical AI development: the assumption that electronic health records provide objective, reliable labels for sensitive conditions like suicidality. The ScAN dataset case study reveals how structural choices in dataset construction—governance constraints, cohort selection methodology, annotation processes, and temporal aggregation—fundamentally alter what an AI system learns to detect. Rather than measuring suicidality as a clinical phenomenon, these datasets encode how clinicians document it under specific institutional conditions.

The study traces how dataset design decisions compound across multiple layers. Single-annotator labeling eliminates inter-rater reliability checks common in social science research. ICD-based cohort selection restricts the sample to documented diagnoses, potentially excluding cases where suicidality was present but undocumented. Hospital-stay-level aggregation erases temporal nuances about when suicidal ideation emerged or resolved. The linguistic analysis demonstrates these choices have real consequences: identical labels subsume clinically distinct cases differing in temporality, negation patterns, and expressed uncertainty.

This matters beyond academic rigor. Clinical NLP systems trained on these datasets may internalize biased operational definitions, perpetuating documentation practices rather than detecting actual clinical presentations. Healthcare organizations deploying suicidality detection tools inherit these embedded assumptions without visibility into their origins. The research suggests that clinical NLP practitioners should treat dataset labels as sociotechnical artifacts requiring interrogation, not empirical ground truth.

The work establishes a methodological imperative for clinical AI: examining assumptions before model development, not after deployment. Future research should explore how different labeling approaches yield different models and whether alternative dataset construction methods better serve clinical detection goals.

• →EHR-based suicidality datasets encode institutional labeling practices rather than objective clinical ground truth.
• →Dataset design choices including single annotators, ICD selection, and temporal aggregation systematically shape what models learn to detect.
• →Identical dataset labels subsume linguistically and clinically heterogeneous cases with different temporal, negation, and uncertainty properties.
• →Clinical NLP systems inherit operational definitions embedded in training data, potentially perpetuating documentation biases in deployment.
• →Researchers should interrogate dataset construction assumptions before interpreting labels or deploying suicidality detection models clinically.