Validating a Reading Assessment Within the Cognitive Diagnostic Assessment Framework: Q-Matrix Construction and Model...

Area of Science:

• Educational Measurement and Assessment
• Cognitive Psychology
• Psychometrics

Background:

• Cognitive Diagnostic Assessment (CDA) offers detailed feedback on student subskills, but research on developing new diagnostic tests, especially for reading, is limited.
• Existing CDA studies often adapt existing large-scale assessments, lacking dedicated test development and robust reliability/validity evidence.
• There is a need for validated diagnostic tools to inform targeted reading instruction.

Purpose of the Study:

• To develop and validate the Diagnostic Chinese Reading Comprehension Assessment (DCRCA) for primary students within the CDA framework.
• To identify optimal cognitive diagnostic models and attribute structures for assessing Chinese reading comprehension.
• To provide evidence of the reliability and validity of the developed DCRCA.

Main Methods:

• Reading attributes were defined through literature review, curriculum standards, expert judgment, and think-aloud protocols.
• Three booklets of reading comprehension items were created for grades 2-6.
• A large-scale study (N=21,466) was conducted, followed by Q-matrix refinement, model-data fit analysis, and comparison of five cognitive diagnostic models (CDMs).

Main Results:

• A six-attribute structure and the G-DINA model demonstrated the best fit for the reading comprehension data.
• The DCRCA exhibited strong diagnostic reliability, construct validity, internal validity, and external validity.
• CDM classifications were found to be reliable, accurate, and useful for educational decision-making.

Conclusions:

• The developed Diagnostic Chinese Reading Comprehension Assessment (DCRCA) is a reliable and valid tool for fine-grained diagnosis of primary students' reading subskills.
• The G-DINA model and a six-attribute structure are suitable for cognitive diagnosis of Chinese reading comprehension.
• The DCRCA provides valuable diagnostic information for students, teachers, and administrators to improve reading instruction and programs.