Counterargument for Critical Thinking as Judged by AI and Humans

A university study of 35 students examined whether writing counterarguments to AI-generated content develops critical thinking skills. Researchers found that student-written counterarguments demonstrated logical reasoning and that six frontier large language models could reliably assess student work using established rubrics, achieving moderate inter-rater reliability (0.33 Gwets AC2) comparable to human assessments.

This intervention study addresses a timely concern in higher education: whether generative AI poses risks of cognitive offloading while simultaneously offering assessment capabilities. The research design directly tackles the tension between AI as a potential academic threat and as a pedagogical tool. By having students write counterarguments to AI-generated thesis statements, the researchers created a structured exercise that appears to combat passive AI consumption and instead foster active critical engagement.

The study's findings carry implications for educational institutions navigating AI integration. The demonstration that students' self-written counterarguments contain logical reasoning suggests that strategic AI use—rather than blanket prohibition—can actually strengthen critical thinking when properly scaffolded. The rubric-based assessment framework spanning focus, logic, content, style, correctness, and references provides educators a replicable methodology.

The moderately aligned inter-rater reliability between AI assessors and human judges (0.33 Gwets AC2, except one model) signals realistic potential for AI-assisted grading at scale, particularly valuable for large enrollment courses facing assessment bottlenecks. However, the moderate rather than high reliability suggests AI assessment still requires human oversight and cannot fully replace educator judgment, particularly for subjective dimensions of critical thinking.

Looking ahead, this research hints at emerging institutional practices where AI becomes integrated into assignment design and assessment workflows rather than treated as an external threat. The work suggests potential for reducing grading burden while maintaining academic rigor, though institutions must carefully balance automation with the human expertise required for nuanced feedback. Further studies with larger sample sizes and diverse institutions would strengthen generalizability.

• →Students writing counterarguments to AI-generated content demonstrate measurable critical thinking logic, countering fears that AI causes cognitive offloading.
• →Large language models can assess student work with moderate reliability comparable to human judges when using clear rubrics, enabling scalable grading.
• →The moderate inter-rater reliability (0.33 Gwets AC2) indicates AI assessment requires human oversight and cannot fully replace educator judgment.
• →Strategic AI use in assignment design—rather than prohibition—may strengthen critical thinking when properly scaffolded with counterargument exercises.
• →Clear rubrics are essential for reliable AI assessment of written work, with 5-point Likert scale evaluations across six dimensions enabling consistent measurement.