Improving reliability estimation in cognitive diagnosis modeling

Area of Science:

• Educational Measurement
• Psychometrics
• Data Science

Background:

• Cognitive diagnosis models (CDMs) are vital for classifying respondents and identifying educational or clinical needs.
• Accurate reliability estimation is essential for valid interpretations of CDM scores.
• Traditional reliability indices often rely on parameter point estimates, potentially leading to overestimated reliabilities due to unaddressed parameter uncertainty.

Purpose of the Study:

• To introduce and evaluate a multiple imputation (MI) procedure for correcting reliability estimation in cognitive diagnosis models (CDMs).
• To compare the accuracy of the MI procedure against traditional reliability estimation methods.

Main Methods:

• A multiple imputation (MI) procedure was developed to integrate out model parameters in estimating posterior distributions.
• A simulation study manipulated attribute structure, CDM model (DINA, G-DINA), test length, sample size, and item quality.
• An empirical analysis was conducted using the Examination for the Certificate of Proficiency in English data, with varying sample sizes.

Main Results:

• The traditional reliability estimation systematically overestimated reliabilities in both simulation and empirical studies.
• The MI procedure provided more accurate reliability estimates compared to the traditional method.
• The positive bias in reliability estimation using point estimates was particularly noted in smaller educational or clinical settings.

Conclusions:

• The proposed multiple imputation (MI) procedure offers a more accurate approach to reliability estimation in cognitive diagnosis models (CDMs).
• Practitioners in smaller settings should be cautious of potential positive bias in reliability estimates derived from model parameter point estimates.
• The study provides R code for the MI procedure, facilitating its adoption in practice.