A Regression Discontinuity Design Framework for Controlling Selection Bias in Evaluations of Differential Item...

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Modeling

Background:

• Differential item functioning (DIF) is crucial for assessing test validity, especially with alternate form accommodations.
• Traditional DIF methods often suffer from selection bias, compromising accuracy.
• Validating test accommodations requires robust DIF analysis to ensure fairness.

Purpose of the Study:

• To propose and evaluate a novel DIF framework using regression discontinuity design to mitigate selection bias.
• To compare the proposed framework against traditional logistic regression in a simulation study.
• To assess Type I error, power, bias, and precision of DIF statistics and effect size estimators.

Main Methods:

• A simulation study was conducted to compare a new DIF framework with traditional logistic regression.
• The regression discontinuity design approach was employed to control for selection bias.
• Key metrics evaluated included Type I error rates, power, bias, and root mean square error.

Main Results:

• The novel DIF framework demonstrated superior control over Type I error rates compared to traditional methods.
• The new framework exhibited minimal bias in effect size estimation.
• However, the proposed method showed lower statistical power and precision.

Conclusions:

• The regression discontinuity-based DIF framework offers a promising approach to reduce selection bias in test accommodation validity studies.
• While effective in controlling errors, further methodological refinement is needed to enhance power and precision for practical application.
• The findings have implications for improving the fairness and validity of standardized testing accommodations.