The Interaction of Ability Differences and Guessing When Modeling Differential Item Functioning With the Rasch Model:...

Area of Science:

• Educational Measurement
• Psychometrics
• Item Response Theory

Background:

• Accurate estimation of differential item functioning (DIF) statistics is essential for item selection and removal in educational testing.
• The Rasch model is commonly used for DIF analysis, but its performance under specific conditions requires careful examination.

Purpose of the Study:

• To investigate potential bias in DIF estimation using the Rasch model when substantial group differences in ability exist.
• To explore the impact of the three-parameter logistic (3PL) model on DIF estimates under conditions of large ability differentials.

Main Methods:

• Simulated data following a three-parameter logistic (3PL) model with varying group ability differences.
• Differential item functioning (DIF) analysis was performed using the Rasch model.
• A mitigation strategy involving coding poorly matched item-examinee encounters as missing data was tested.

Main Results:

• The Rasch model introduced significant bias in DIF estimates when large group ability differences were present, particularly with 3PL data.
• Difficult non-DIF items appeared to favor the focal group, while easy non-DIF items favored the reference group under these conditions.
• Bias in DIF effect sizes was observed, but these effects were reduced when data from examinees with low ability relative to item difficulty were treated as missing.

Conclusions:

• Measurement practitioners should exercise caution when interpreting Rasch model-based DIF estimates in the presence of large ability group differences and potential guessing.
• The study highlights the distorting effect of guessing on item response functions across different ability distributions.
• Excluding data from examinees poorly matched to item difficulty can mitigate bias in DIF estimation under specific conditions.