A Comparison of Differential Item Functioning Detection Methods in Cognitive Diagnostic Models.

Area of Science:

• Educational Measurement and Psychometrics
• Cognitive Diagnostic Models
• Statistical Modeling

Background:

• Cognitive diagnostic models (CDMs) are discrete latent variable models used across education and psychology.
• Detecting differential item functioning (DIF) is vital for ensuring the fairness and validity of CDMs.
• Existing DIF detection methods require rigorous evaluation within the context of CDMs.

Purpose of the Study:

• To investigate the performance of various statistical methods for detecting DIF in cognitive diagnostic tests.
• To compare the Type I error rates and statistical power of different DIF detection approaches under varying conditions.

Main Methods:

• A Monte Carlo simulation study was employed to evaluate six DIF detection methods.
• Methods included Mantel-Haenszel (MH), logistic regression (LR), and Wald tests utilizing different covariance matrices (W_d, W_XPD, W_Obs, W_Sw).
• Simulated data varied factors such as item quality and DIF type (uniform and non-uniform).

Main Results:

• The cross-product information (W_XPD) and logistic regression (LR) methods demonstrated superior control of Type I error rates.
• Under uniform DIF, W_XPD, W_Obs, and W_Sw showed comparable or superior power to MH and LR with high/medium item quality, but less power with low item quality.
• Under non-uniform DIF, W_XPD, W_Obs, and W_Sw exhibited comparable or higher power than LR.

Conclusions:

• W_XPD and LR methods are recommended for their robust Type I error control in DIF detection within CDMs.
• The choice of covariance matrix (W_XPD, W_Obs, W_Sw) impacts statistical power, particularly concerning item quality and DIF type.
• Further research may refine these methods for optimal DIF detection in diverse cognitive diagnostic testing scenarios.