From Test-taking to Cognitive Scaffolding: A Pedagogical Diagnostic Benchmark for LLMs on English Standardized Tests

Researchers introduce ESTBook, a pedagogical diagnostic benchmark containing 10,576 multimodal questions across five major English standardized tests, designed to evaluate whether large language models can exhibit faithful reasoning and identify student misconceptions rather than just achieving binary accuracy scores. The framework moves beyond traditional test-taking benchmarks by enriching questions with cognitive reasoning trajectories and distractor rationales, enabling better assessment of LLM capabilities as educational tutoring tools.

The introduction of ESTBook represents a meaningful shift in how the AI research community evaluates large language models within educational contexts. Rather than measuring success through simple right-or-wrong metrics, this work acknowledges that effective AI tutors must demonstrate reasoning transparency, pedagogical strategy articulation, and the ability to diagnose why students select incorrect answers. This paradigm reflects growing recognition that LLM integration into education demands sophistication beyond raw accuracy metrics.

This research addresses a critical gap in current AI evaluation methodologies. Standardized test benchmarks have historically prioritized outcome measurement, but educational applications require understanding the reasoning paths that lead to answers and the cognitive misconceptions that produce common errors. By formalizing reasoning trajectories and distractor rationales, ESTBook provides infrastructure for diagnosing these nuanced failure modes.

For the EdTech and AI sectors, this work has practical implications. Developers building AI tutoring systems now have a more rigorous framework for measuring pedagogical effectiveness, potentially accelerating the adoption of LLM-powered educational tools. The multimodal nature of the benchmark—spanning 29 task types across five exams—creates a comprehensive testing ground that better reflects real-world classroom complexity than narrower datasets.

Moving forward, the success of diagnostic frameworks like ESTBook will likely influence how educational institutions evaluate AI tutors before deployment. Research building on this work may establish new standards where pedagogical reasoning quality becomes as important as accuracy metrics, reshaping investment and development priorities across EdTech and AI sectors.

• →ESTBook benchmark contains 10,576 multimodal questions with formalized reasoning trajectories enabling pedagogical diagnosis beyond binary accuracy
• →Research demonstrates that identifying cognitive trajectories helps mitigate performance gaps and improves AI tutoring effectiveness
• →Framework models English standardized test problem-solving as cognitive traversal rather than simple right-or-wrong outcomes
• →Diagnostic approach reveals LLM capabilities in explaining solution strategies and identifying specific student misconceptions
• →Study establishes new evaluation standards prioritizing pedagogical reasoning quality alongside accuracy for educational AI applications