Implementing a Standardized Effect Size in the POLYSIBTEST Procedure.

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Modeling

Background:

• Differential item functioning (DIF) analysis is crucial for test fairness.
• Existing methods for polytomous data often lack standardized effect sizes and clear guidelines.
• The POLYSIBTEST procedure offers a framework for analyzing polytomous item data.

Purpose of the Study:

• To implement and evaluate standardized effect size guidelines for polytomous data within the POLYSIBTEST procedure.
• To compare the performance of a new standardized effect size with existing unstandardized and standardized methods.
• To provide practical tools for researchers analyzing differential item functioning in polytomous data.

Main Methods:

• Two simulation studies were conducted to assess effect size heuristics.
• The first simulation identified unstandardized heuristics for moderate and large DIF in polytomous data (3-7 response options).
• The second simulation compared standardized effect sizes (Weese, Zwick et al.) and unstandardized procedures (Gierl, Golia) using true-positive and false-positive rates.

Main Results:

• All tested procedures maintained false-positive rates below significance levels for moderate and large DIF.
• Weese's standardized effect size demonstrated robustness to sample size and yielded higher true-positive rates than Zwick et al. and Golia.
• Weese's method flagged fewer items with negligible DIF compared to Gierl's criterion.

Conclusions:

• The proposed standardized effect size for polytomous data is practical and interpretable for practitioners.
• It offers advantages in identifying differential item functioning across various response scales.
• This contributes to more accurate and accessible psychometric analysis of polytomous items.