Automated Coding of Communication Data Using ChatGPT: Consistency Across Subgroups

Research demonstrates that ChatGPT can code communication data with accuracy comparable to human raters while maintaining consistency across different demographic groups including gender and racial/ethnic categories. The study introduces three evaluation checks for assessing subgroup consistency in LLM-based coding systems for large-scale collaboration assessments.

• →ChatGPT achieves human-comparable accuracy when coding communication data using direct instruction with coding rubrics.
• →The AI system maintains consistent performance across gender and racial/ethnic demographic groups, matching human rater patterns.
• →Three new evaluation checks were developed to assess subgroup consistency in LLM-based coding systems.
• →Results support the viability of using ChatGPT for large-scale assessments of collaboration and communication.
• →The research addresses critical fairness concerns about AI bias across different demographic subgroups.