Controlling the False Discovery Rate in DIF Detection With e-Values: Evidence From Multidimensional and Testlet...

Area of Science:

• Psychometrics
• Statistical modeling
• Educational measurement

Background:

• Traditional p-value methods for Differential Item Functioning (DIF) detection face limitations when statistical assumptions are violated.
• Issues like multidimensionality, local item dependence, and extreme sample sizes can compromise the accuracy of p-value-based approaches.

Purpose of the Study:

• To apply e-value-based false discovery rate (FDR) control for DIF detection, offering an alternative to p-value methods.
• To evaluate the performance of e-BH procedures against classical methods under various model misspecification scenarios.

Main Methods:

• Conducted two simulation studies evaluating K-fold and Multisplit likelihood-ratio e-values with e-BH procedures.
• Simulated conditions included multidimensional contamination and testlet-based local dependence.
• Applied e-BH to Progress in International Reading Literacy Study (PIRLS) data for empirical validation.

Main Results:

• E-BH demonstrated superior and more stable control of Type I error, FDR, and family-wise error rate (FWER) compared to Benjamini-Hochberg (BH) and Holm procedures.
• E-BH maintained lower false-positive rates even under severe model misspecification, with competitive Type II error rates.
• Classical p-value methods showed increased Type I error with larger sample sizes, while e-BH maintained stable control due to model-agnostic calibration.

Conclusions:

• E-value-based FDR control (e-BH) is a robust and effective tool for DIF detection in complex assessment contexts.
• E-BH provides more reliable and sustainable DIF flagging than traditional p-value approaches, particularly when model assumptions are violated.
• The model-agnostic calibration of e-BH ensures stable error control across varying sample sizes and model misspecifications.