From automated scoring to learning diagnosis: a mechanism study of AI-supported formative assessment in English...

AI-supported writing assessment can provide rapid scores and issue-level comments, yet its formative value depends on how students and teachers interpret and act on that information. This study examined this process in a university English writing course.

Two pre-existing intact classes comprising 96 students (48 per class) completed three writing-feedback-revision cycles. The Control class received AI scores/standard feedback, routine teacher checking, and revision. The Diagnostic Review class received DeepSeek-R1 issue-level diagnoses, completed an interpretation sheet, revised independently, underwent structured teacher review and reassessment, and recorded a reflection. Measures included diagnostic accuracy, feedback actionability, feedback comprehension, self-assessment accuracy, revision quality, reflection quality, misunderstanding codes, and teacher review activity. Mixed-effects models with student random intercepts were used for analysis.

The Diagnostic Review class showed higher scores for diagnostic accuracy, actionability, comprehension, revision quality, and self-assessment accuracy. Group-by-cycle interactions were significant for diagnostic accuracy, comprehension, revision quality, and self-assessment accuracy, whereas the actionability interaction was not. Feedback comprehension showed the strongest bivariate association with revision quality. Exploratory indirect associations through actionability and comprehension remained positive after adjustment for condition and cycle using student-cluster bootstrap confidence intervals. Teacher records documented confirmation, correction, supplementation, and re-explanation, with greater review time in the Diagnostic Review class.

The findings indicate that the formative value of AI-supported writing assessment extends beyond automated diagnosis to the subsequent processes of student interpretation, revision, and teacher review. Repeated classroom assessment may therefore benefit from integrating AI-generated diagnostic feedback with structured learner reflection and context-sensitive teacher judgment.