Refining effect size measures and classification for differential item functioning: Toward unified guidelines across...

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Analysis

Background:

• Differential Item Functioning (DIF) analysis is crucial for detecting bias in test items.
• Evaluating the practical significance of DIF requires appropriate effect size measures.
• Existing effect size measures and classification guidelines have limitations.

Purpose of the Study:

• To review and evaluate existing DIF effect size measures and classification guidelines.
• To introduce a refinement of area-based effect size measures.
• To propose revised and new cut-off values for improved DIF assessment.

Main Methods:

• Review of existing effect size measures for Mantel-Haenszel, SIBTEST, and logistic regression.
• Conducting a simulation study to assess properties and comparative performance of measures.
• Developing and validating refined area-based effect size measures.

Main Results:

• Commonly used effect size measures show inconsistent classifications and underestimate DIF magnitude.
• Some measures exhibit strong dependence on study design factors.
• Revised cut-off values and new area-based measures demonstrate improved performance.

Conclusions:

• Existing DIF effect size measures require careful usage and revised classification guidelines.
• Proposed refined area-based measures and unified cut-off values enhance DIF practical significance evaluation.
• Implementation in R software facilitates the application of these improved methods.