Using Interpretable Machine Learning for Differential Item Functioning Detection in Psychometric Tests.

Area of Science:

• Psychometrics
• Machine Learning
• Educational Measurement

Background:

• Test unfairness arises from demographic influences on residual variances in psychometric models.
• Existing methods struggle with complex relationships between response patterns and demographic attributes.

Purpose of the Study:

• To present a novel method combining psychometrics and machine learning for investigating test fairness.
• To measure the predictive importance of test items and latent ability scores for demographic characteristics.

Main Methods:

• Utilized random forests to predict demographic attributes from test responses.
• Conducted simulation studies to assess method performance under various conditions.
• Applied the method to an elementary school reading comprehension test to predict migration background.

Main Results:

• The novel method reliably detects unfair items, comparable to Mantel-Haenszel statistics and logistic regression.
• The approach generalizes effectively to multidimensional scales.
• One item in the reading comprehension test was identified as unfair based on proposed criteria.

Conclusions:

• The proposed machine learning-based method offers a robust approach to identifying test unfairness.
• This technique enhances the ability to detect differential item functioning in educational assessments.
• Findings support the use of machine learning for improving the fairness and validity of standardized tests.