Faster, Cheaper, More Accurate: Specialised Knowledge Tracing Models Outperform LLMs

Research comparing Knowledge Tracing (KT) models to Large Language Models (LLMs) for predicting student responses found that specialized KT models significantly outperform LLMs in accuracy, speed, and cost-effectiveness. The study demonstrates that domain-specific models are superior to general-purpose LLMs for educational prediction tasks, with LLMs being orders of magnitude slower and more expensive to deploy.

• →Specialized Knowledge Tracing models outperform LLMs in accuracy and F1 scores for predicting student responses to educational questions.
• →LLMs are orders of magnitude slower in inference speed compared to domain-specific KT models.
• →Deployment costs for LLMs are significantly higher than specialized educational models.
• →The research challenges the notion that LLMs should be used as universal solutions for all prediction tasks.
• →Domain-specific models remain crucial for educational technology applications despite the rise of general-purpose AI.