Using a Multidimensional IRT Framework to Better Understand Differential Item Functioning (DIF): A Tale of Three DIF...

Area of Science:

• Psychometrics
• Educational Measurement
• Item Response Theory

Background:

• Differential item functioning (DIF) is theoretically linked to multidimensionality in assessment data.
• Existing research suggests DIF detection thresholds may be influenced by secondary ability distributions.

Purpose of the Study:

• To quantify the necessary difference in secondary ability distributions for DIF detection.
• To investigate the impact of varying secondary dimension mean differences and inter-dimensional correlations on DIF detection.

Main Methods:

• Simulated two-dimensional binary data using a compensatory multidimensional item response theory (MIRT) model.
• Systematically varied mean differences on the second dimension between reference and focal groups.
• Employed SIBTEST, Mantel-Haenszel, and logistic regression for DIF detection.

Main Results:

• DIF was detected even with minimal mean differences on the secondary dimension, smaller than previously considered.
• Small mean differences (0.25) on the secondary dimension consistently led to statistically significant subtest score differences between groups.

Conclusions:

• The sensitivity of DIF detection procedures is high, identifying bias even with subtle group differences in secondary abilities.
• Assessment of secondary dimensions is crucial for ensuring test fairness and accurate interpretation of examinee performance.