Can LLMs Model Incorrect Student Reasoning? A Case Study on Distractor Generation

Research from arXiv examines how large language models generate multiple-choice distractors for educational assessments by modeling incorrect student reasoning. The study finds LLMs surprisingly align with educational best practices, first solving problems correctly then simulating misconceptions, with failures primarily occurring in solution recovery and candidate selection rather than error simulation.

• →LLMs demonstrate surprising alignment with established educational best practices when generating multiple-choice distractors.
• →Models typically follow a three-step process: solve correctly first, simulate misconceptions, then select distractors.
• →Primary failure modes occur in solution recovery and response selection rather than in simulating student errors.
• →Providing the correct solution in prompts improves alignment with human-authored distractors by 8%.
• →The research provides structured insights into LLMs' ability to model incorrect reasoning for educational applications.