CODE-GEN: A Human-in-the-Loop RAG-Based Agentic AI System for Multiple-Choice Question Generation

Researchers developed CODE-GEN, a human-in-the-loop AI system that uses retrieval-augmented generation to create multiple-choice programming questions for educational purposes. The system achieved 79.9% to 98.6% success rates across seven pedagogical dimensions when evaluated by subject-matter experts, demonstrating strong performance in computational verification tasks while still requiring human expertise for complex instructional design.

• →CODE-GEN uses an agentic AI architecture with separate Generator and Validator agents for creating and assessing educational content quality.
• →Human evaluation involving six experts and 288 AI-generated questions showed success rates between 79.9% and 98.6% across pedagogical dimensions.
• →The system excels at computationally verifiable tasks like code validity and concept alignment but struggles with complex instructional design.
• →Human expertise remains essential for creating meaningful distractors and providing high-quality educational feedback.
• →The research provides insights for optimally allocating human and AI resources in educational content generation.