Comparing the Impact of Pedagogy-Informed Custom and General-Purpose GAI Chatbots on Students' Science Problem-Solving Processes and Performance Using Heterogeneous Interaction Network Analysis

Researchers compared custom pedagogy-informed AI chatbots with general-purpose chatbots like ChatGPT for science education, finding that custom chatbots using Socratic questioning methods increased student cognitive engagement and reduced cognitive offloading. The study analyzed 3,297 student-chatbot dialogues from 48 secondary school students, showing higher interaction intensity with custom chatbots despite similar problem-solving performance outcomes.

• →Custom AI chatbots designed with Socratic questioning methods significantly increased student interaction intensity and cognitive engagement compared to general-purpose chatbots.
• →Students were more likely to follow guidance and reflect when using custom chatbots, while they tended to request direct command execution from general-purpose chatbots like ChatGPT.
• →Despite higher engagement, both chatbot types produced similar problem-solving performance outcomes in terms of solution quality.
• →The study provides evidence that custom chatbots reduce cognitive offloading, a common issue with general-purpose AI assistants in education.
• →The research offers insights for designing AI chatbots specifically for educational applications rather than relying on general-purpose models.