AgentCAT: Simulating Computerized Adaptive Testing via Multi-Agent Large Language Models

AgentCAT is a new Large Language Model-based multi-agent simulation system designed to improve computerized adaptive testing (CAT) by creating a high-fidelity benchmarking environment. The framework addresses limitations of existing CAT research by simulating the complete dynamic assessment process through three specialized agents: an examinee agent with reasoning capabilities, a selection agent for exercise optimization, and a supervisor ensuring validity.

AgentCAT represents a meaningful advancement in educational assessment technology by moving beyond static offline approaches to dynamic, multi-agent simulation. Traditional CAT systems suffer from fragmented research that isolates individual components like item selection or diagnostic feedback, preventing holistic optimization of the testing experience. The AgentCAT framework tackles this by introducing LLM-powered agents that simulate realistic examinee behavior, intelligently select questions, and oversee the assessment process—creating a closed-loop system that mirrors actual human testing dynamics.

The research addresses a genuine bottleneck in educational technology: existing datasets contain partial labels and sparse information, limiting researchers' ability to optimize for real-world testing scenarios. By constructing a synthetic but high-fidelity simulation environment, AgentCAT enables researchers to study interactions that rarely occur in static logs while maintaining pedagogical validity. The three-tier agent architecture—combining cognitive modeling, strategic selection with knowledge graph exploration, and robust convergence validation—demonstrates sophisticated system design.

For educational institutions and edtech companies, AgentCAT offers both theoretical insights and practical value. The validation across macro-level ability convergence and micro-level interaction logic suggests the system could improve assessment accuracy and learning outcomes when deployed. Educational software developers could integrate similar multi-agent approaches to create more responsive, personalized testing platforms. The framework's data sparsity resilience particularly matters for scaling adaptive testing to underserved educational markets with limited assessment data.

• →AgentCAT uses multi-agent LLM simulation to create realistic computerized adaptive testing benchmarks, addressing limitations of static offline data.
• →Three specialized agents—examinee, selection, and supervisor—work together to optimize dynamic assessment while balancing difficulty adaptation and instructional coherence.
• →The framework validated successfully on real-world datasets, demonstrating effective ability estimation and pedagogical alignment with human teaching intuition.
• →LLM-based simulation enables research on assessment interactions that rarely appear in existing datasets, overcoming data sparsity challenges.
• →Results suggest practical applications for educational technology developers seeking to improve personalized assessment systems at scale.