Quantifying 'problematic' DIF within an IRT framework: application to a cancer stigma index

Area of Science:

• Psychometrics
• Item Response Theory (IRT)
• Health Measurement

Background:

• Differential Item Functioning (DIF) detection is crucial in Item Response Theory (IRT) to ensure measurement invariance.
• However, statistically significant DIF may not always indicate practical or clinical importance, necessitating refined evaluation methods.

Purpose of the Study:

• To introduce and evaluate two novel metrics: weighted area between the expected score curves (wABC) and difference in expected a posteriori scores (dEAP).
• These metrics aim to discern clinically meaningful DIF among items flagged by traditional IRT DIF statistics.

Main Methods:

• The study computed wABC and dEAP metrics.
• Data from a 27-item cancer stigma index administered to Arabic, English, and Mandarin speakers across Jordan, Egypt, and China were analyzed.
• Item Response Theory-based Wald chi-square DIF statistics were calculated, followed by p-value adjustments and evaluation using wABC and dEAP.

Main Results:

• Twenty statistically significant DIF comparisons were identified post-adjustment.
• DIF comparisons with wABC values greater than 0.3 were considered potentially problematic.
• The dEAP metric provided insights into DIF impact but was less consistent in identifying problematic items compared to wABC.

Conclusions:

• wABC and dEAP serve as effective DIF effect size indicators.
• These metrics significantly enhance IRT DIF evaluation by differentiating statistically significant DIF from practically significant DIF.