Examining the Impact of Differential Item Functioning on Classification Accuracy in Cognitive Diagnostic Models
Justin Paulsen1, Dubravka Svetina1, Yanan Feng1
1Indiana University, Bloomington, USA.
Applied Psychological Measurement
|June 16, 2020
Summary
Differential item functioning (DIF) impacts cognitive diagnostic models (CDMs). While attribute classification accuracy is robust, profile classification accuracy decreases with DIF, especially with unequal groups and simple structure items.
Area of Science:
- Educational Measurement and Psychometrics
- Cognitive Diagnostic Models (CDMs)
- Differential Item Functioning (DIF)
Background:
- Cognitive diagnostic models (CDMs) are increasingly used in educational research for detailed diagnostic information on examinee strengths and weaknesses.
- Understanding the impact of differential item functioning (DIF) is crucial for the appropriate use and interpretation of CDMs.
- Limited research exists on how DIF affects classification accuracy within CDMs.
Purpose of the Study:
- To investigate the influence of DIF magnitudes and types on classification accuracy in cognitive diagnostic models.
- To examine the interaction of DIF with CDM item types, group distributions, and sample sizes.
- To provide a reference for practitioners regarding the impact of DIF on attribute- and profile-level classification accuracy.
Main Methods:
- A simulation study was conducted to systematically evaluate the effects of various DIF conditions.
- The study manipulated DIF magnitudes, DIF types, item structures, group distributions, and sample sizes.
- Classification accuracy at both the attribute and profile levels was assessed under these simulated conditions.
Main Results:
- Attribute-level classification accuracy demonstrated robustness to large magnitudes of DIF across most simulated conditions.
- Profile-level classification accuracy was negatively impacted by the presence of DIF.
- Unequal group distributions and DIF on simple structure items significantly reduced classification accuracy.
Conclusions:
- While attribute mastery can be accurately classified even with substantial DIF, overall profile classification is more vulnerable.
- Practitioners should be cautious when interpreting profile classifications from CDMs, especially in settings with unequal group sizes or items exhibiting DIF.
- Future research should further explore the practical implications of these findings and develop strategies to mitigate DIF's negative effects on classification accuracy.
Related Concept Videos
01:27Diagnostic and Statistical Manual of Mental Disorders (DSM)
693
The Diagnostic and Statistical Manual of Mental Disorders (DSM) serves as the primary classification system for mental health disorders, providing standardized diagnostic criteria for clinicians and researchers. First published by the American Psychiatric Association (APA) in 1952, the DSM has undergone several revisions to reflect evolving psychiatric understanding. The fifth edition, DSM-5, released in 2013, introduced key updates that expanded diagnostic categories and modified diagnostic...
693
01:15Receiver Operating Characteristic Plot
424
A ROC (Receiver Operating Characteristic) plot is a graphical tool used to assess the performance of a binary classification model by illustrating the trade-off between sensitivity (true positive rate) and specificity (false positive rate). By plotting sensitivity against 1 - specificity across various threshold settings, the ROC curve shows how well the model distinguishes between classes, with a curve closer to the top-left corner indicating a more accurate model. The area under the ROC curve...
424