Enhancing Precision in Predicting Magnitude of Differential Item Functioning: An M-DIF Pretrained Model Approach.

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Modeling

Background:

• Existing differential item functioning (DIF) detection methods lack consistent effect size definitions and adequate consideration of testing conditions.
• Inconsistent DIF magnitude estimation hinders accurate interpretation and application in large-scale assessments.

Purpose of the Study:

• To introduce a unified M-DIF model for a consistent and quantitative definition of DIF magnitude.
• To develop a robust model that incorporates various DIF detection methods and test conditions.
• To enhance the generalizability and applicability of DIF analysis across different datasets and testing scenarios.

Main Methods:

• Developed the unified M-DIF model defining DIF magnitude as the difference in item difficulty parameters between reference and focal groups.
• Employed a pretrained approach using a large training dataset (144 combinations of test conditions, 144,000 items, 29 metrics) with XGBoost modeling.
• Validated the model's performance against a baseline model using root mean square error (RMSE) and BIAS metrics under consistent and inconsistent test conditions.

Main Results:

• The M-DIF model significantly outperformed the baseline model in both validation sets, demonstrating superior accuracy under consistent and inconsistent test conditions.
• Across 360 test condition combinations, the M-DIF model exhibited lower RMSE in 99.2% of cases, highlighting its robustness and reliability.
• An empirical example confirmed the practical feasibility and effectiveness of implementing the M-DIF model in real-world assessment scenarios.

Conclusions:

• The unified M-DIF model provides a standardized and accurate method for quantifying DIF magnitude, addressing limitations of existing approaches.
• The model's robustness across varied testing conditions makes it a valuable tool for improving the quality and fairness of educational and psychological assessments.
• The pretrained approach ensures model generalizability, allowing for direct application to new data and facilitating more reliable DIF detection.