Detecting Differential Item Functioning Using Multiple-Group Cognitive Diagnosis Models.

Area of Science:

• Educational Measurement
• Psychometrics
• Cognitive Psychology

Background:

• Students may use different strategies to solve problems, impacting test item performance.
• Detecting differential item functioning (DIF) is crucial for fair and accurate assessments.
• Existing DIF detection methods may be sensitive to item quality and group differences.

Purpose of the Study:

• To propose a multiple-group cognitive diagnosis model accounting for varied attribute usage.
• To evaluate the performance of likelihood ratio (LR) and Wald tests for DIF detection under this model.
• To develop and assess a forward anchor item search procedure for identifying invariant items.

Main Methods:

• Developed a multiple-group cognitive diagnosis model allowing for distinct or varied attribute use (conjunctive, disjunctive, compensatory).
• Systematically investigated the Type I error rates and power of LR and Wald tests for DIF detection.
• Proposed and implemented a forward anchor item search algorithm to select stable anchor items across groups.

Main Results:

• The LR and Wald tests combined with the forward anchor item search demonstrated better calibrated Type I error rates compared to standard methods.
• This improvement was particularly notable when dealing with lower-quality test items.
• The proposed methods were illustrated using real-world data analysis.

Conclusions:

• The proposed multiple-group cognitive diagnosis model provides a framework for understanding diverse problem-solving strategies.
• The forward anchor item search procedure enhances the reliability of DIF detection, especially in challenging scenarios.
• These advancements contribute to more equitable and precise educational assessments.