Beyond retrofitting: validating a purpose-built reading diagnostic assessment through a cognitive diagnostic approach

Diagnostic assessment, which aims to identify the strengths and weaknesses of learners' cognitive abilities for targeted remediation, has gained momentum in recent years. This study applied a cognitive diagnostic approach to validate the UDig Reading Test, designed to diagnose the reading skills of tertiary-level English as a foreign language (EFL) learners in China.

Using a mixed-methods design, test-takers' think-aloud protocols, expert judgments, and cognitive diagnostic modeling were integrated to examine the accuracy of the predefined item-subskill associations. Large-scale performance data from 642 test-takers were analyzed to estimate their reading subskill mastery patterns.

Results showed that the predefined item-subskill associations were generally supported by empirical evidence. In particular, search reading was empirically confirmed as a targeted subskill in the expeditious reading section, despite its absence from the initial test specifications. Subskill prevalence analysis identified the subskill of understanding the structure and cohesion of a passage as the most challenging one, providing evidence that may inform targeted instructional support.

This study responds to persistent calls for research into truly diagnostic assessments rather than retrofitting existing tests and demonstrates the value of the cognitive diagnostic approach in enhancing the diagnostic precision of reading assessment. It also outlines a research agenda for developing cognitive diagnostic assessment from inception so as to fully realize its diagnostic potential.