Automated Coding of Communication Data Using ChatGPT: Consistency Across Subgroups

Research demonstrates that ChatGPT can code communication data with accuracy comparable to human raters while maintaining consistency across different demographic groups including gender and racial/ethnic categories. The study introduces three evaluation checks for assessing subgroup consistency in LLM-based coding systems for large-scale collaboration assessments.