Improving Accuracy and Usage by Correctly Selecting: The Effects of Model Selection in Cognitive Diagnosis...

Area of Science:

• Educational Measurement and Psychometrics
• Computerized Adaptive Testing
• Cognitive Diagnosis Models

Background:

• Model selection is a critical decision in calibrating item banks for adaptive algorithms.
• Cognitive Diagnosis Computerized Adaptive Testing (CD-CAT) faces challenges due to numerous available models.
• The performance of adaptive algorithms is highly sensitive to calibration decisions.

Purpose of the Study:

• To investigate the utility of model selection indices for enhancing CD-CAT performance.
• To determine if model selection can optimize adaptive testing algorithms.
• To evaluate the impact of calibration sample size, Q-matrix complexity, and item bank length.

Main Methods:

• A simulation study was conducted to compare different model estimation strategies.
• Compared results based on true item parameters, general models, single reduced models, and combined models selected by fit indices.
• Investigated the influence of calibration sample size, Q-matrix complexity, and item bank length.

Main Results:

• Fitting a single reduced or a general model does not typically yield optimal results.
• Combining models selected via fit indices produced results closest to those using true item parameters.
• Model selection indices improve classification accuracy and balance item bank usage.

Conclusions:

• Model selection indices are effective in improving CD-CAT performance.
• Combining appropriate models leads to more accurate classifications and efficient testing.
• The developed 'cdcatR' R package facilitates practical implementation of these findings.