Combining CAT with cognitive diagnosis: a weighted item selection approach

Area of Science:

• Educational Measurement
• Psychometrics
• Computer Science

Background:

• Computerized adaptive testing (CAT) enhances measurement precision by adapting item selection based on prior responses.
• Current CAT systems often lack robust diagnostic capabilities for detailed student knowledge assessment.
• Cognitive diagnostic testing aims to classify student knowledge status (α) alongside trait estimation (θ).

Purpose of the Study:

• To develop and evaluate item selection strategies for cognitive diagnostic tests within a CAT framework.
• To identify an algorithm that efficiently estimates latent traits (θ) and accurately classifies knowledge status (α).
• To integrate diagnostic criteria as constraints within an item selection process.

Main Methods:

• A single-stage item selection method using the maximum priority index is proposed.
• Diagnostic criteria are treated as constraints to guide item selection.
• Priority functions are developed based on information measures (e.g., Kullback-Leibler information) and Q-matrix data.
• An extensive simulation study compares different methods, considering item exposure control.

Main Results:

• The information-based item selection method demonstrated superior cognitive diagnosis classification rates.
• The proposed method also achieved more accurate estimation of the latent trait (θ).
• The maximum priority index approach effectively integrates diagnostic feedback generation.

Conclusions:

• The developed item selection strategy offers an effective dual-purpose approach for CAT in cognitive diagnosis.
• Information-based priority functions are recommended for enhanced diagnostic accuracy and trait estimation.
• The method provides a straightforward way to generate cognitive diagnostic feedback within CAT systems.