Differential item functioning magnitude and impact measures from item response theory models

Area of Science:

• Psychometrics
• Educational Measurement
• Statistics

Background:

• Differential item functioning (DIF) is crucial for ensuring fairness in assessments.
• Existing methods for quantifying DIF magnitude and impact vary.
• Item response theory (IRT) provides a robust framework for DIF analysis.

Purpose of the Study:

• To present and review measures of DIF magnitude (item level) and impact (scale level).
• To demonstrate the similarities among various DIF indices.
• To introduce software for calculating and comparing DIF statistics.

Main Methods:

• Review of existing magnitude measures based on item response theory and group differences in expected item scores.
• Review of existing impact measures based on group differences in expected scale scores.
• Demonstration of mathematical similarities between magnitude and impact indices.

Main Results:

• Magnitude measures quantify item-level effects, while impact measures assess scale-level differences between groups.
• Several software packages exist for DIF analysis, with varying capabilities.
• New software is presented that integrates multiple DIF statistics and their relationships.

Conclusions:

• A comprehensive understanding of DIF magnitude and impact measures is essential for test fairness.
• The presented software facilitates a more convenient and integrated approach to DIF analysis.
• Further research can leverage these integrated tools for robust psychometric evaluations.