Consensus Among Differential Item Functioning Effect Size Measures: A Simplified Approach to Reporting Effect Size

Area of Science:

• Psychometrics
• Educational Measurement
• Statistics

Background:

• Differential item functioning (DIF) can create score disparities across subpopulations.
• Effect size measures are crucial for assessing the practical significance of DIF.
• Previous research has not evaluated the agreement among various DIF effect size measures.

Purpose of the Study:

• To investigate the relationships among 15 DIF effect size measures.
• To provide recommendations for reporting DIF effect sizes in 1PL, 2PL, and GRM models.
• To enhance the detection of DIF and its practical implications.

Main Methods:

• Reviewed 15 different DIF effect size measures.
• Conducted simulations across 1PL, 2PL, and GRM models.
• Analyzed measure agreement and bias under varying sampling conditions.

Main Results:

• Identified three key effect size measures (Mantel-Haenszel Delta, McFadden's Pseudo R-squared, and a model-specific signed DIF measure) offering unique information and minimal bias.
• Determined optimal signed DIF measures vary by model: Signed Area under the curve (1PL), Standardized P-difference (2PL), and Signed Item Difference (GRM).

Conclusions:

• Recommends reporting Mantel-Haenszel Delta, McFadden's Pseudo R-squared, and a specific signed DIF measure for comprehensive DIF analysis.
• The proposed reporting strategy improves DIF detection and addresses measurement bias.
• Provides applied researchers with practical guidance for reporting DIF effect sizes.