Detecting Knowledge Gaps from Conversational AI Interactions Using Curriculum Prerequisite Graphs

Researchers developed a pipeline using GPT-4 and few-shot learning to map student questions from conversational AI teaching assistants to curriculum topics, achieving 80% classification accuracy. The classified question data correlates with student-reported difficulty levels, demonstrating that AI interaction logs can serve as diagnostic tools for identifying knowledge gaps and informing instructional design.

This research addresses a significant untapped resource in educational technology: the diagnostic potential of conversational AI interaction logs. Traditional methods for identifying student knowledge gaps rely on assessments, surveys, and attendance patterns—all requiring explicit student participation. By automatically classifying thousands of questions asked to AI teaching assistants and mapping them onto curriculum structure, educators gain passive, real-time signals about which topics genuinely challenge students. The 80% classification accuracy across 43 distinct topics represents a substantial practical achievement, validated by the correlation between question volume and independent student surveys (p=0.008), providing strong evidence that the system captures meaningful pedagogical signals rather than noise. This approach scales naturally with course size, as larger cohorts generate richer interaction data. The prerequisite knowledge graph architecture enables more sophisticated analysis—beyond identifying difficult topics, the system could eventually predict cascading knowledge deficits when students struggle with foundational concepts. For educational institutions, this methodology transforms passive AI tutor interactions into actionable curriculum intelligence. Instructors can identify which topics need reinforcement, adjust pacing, or redesign explanations based on empirical evidence rather than intuition. The convergent validity with student surveys strengthens claims that this represents genuine difficulty rather than simple question popularity. Looking forward, similar approaches could apply to enterprise training systems, online learning platforms, and corporate onboarding programs. The challenge remains scaling the prerequisite graph extraction process and handling domain-specific terminology across different course types and disciplines.

• →Conversational AI interaction logs can be automatically mapped to curriculum topics with 80% accuracy using few-shot classification and GPT-4-extracted knowledge graphs.
• →Question volume from AI assistants correlates significantly with student-reported difficulty (rho=0.491, p=0.008), validating the diagnostic signal.
• →This passive monitoring approach requires no additional student effort or assessment overhead compared to traditional knowledge gap detection.
• →Prerequisite knowledge graphs enable curriculum-grounded analysis that could reveal cascading knowledge deficits and foundational concept gaps.
• →The methodology scales naturally with course size and could apply across online learning platforms, corporate training, and enterprise education systems.