Explaining Person-by-Item Responses using Person- and Item-Level Predictors via Random Forests and Interpretable...

Area of Science:

• Psychometrics
• Machine Learning
• Educational Measurement

Background:

• Item response models (IRTs) traditionally struggle with complex predictor interactions.
• Random Forests (RF) excel at capturing nonlinearities but lack interpretability.
• Existing models may not fully account for person- and item-level predictor effects simultaneously.

Purpose of the Study:

• To develop and evaluate a hybrid model (EIRM-RF) combining Explanatory Item Response Models (EIRM) and Random Forests (RF).
• To improve the explanation of item responses by modeling nonlinear and interaction effects.
• To assess the performance of interpretable machine learning (ML) methods within this hybrid framework.

Main Methods:

• Integration of RF predicted values as a predictor within an EIRM framework (EIRM-RF).
• Application of interpretable ML techniques: feature importance, partial dependence plots, accumulated local effect plots, and the H-statistic.
• Illustration with an empirical dataset on reading comprehension differences (digital vs. paper) and simulation studies for model comparison.

Main Results:

• The EIRM-RF model demonstrates superior performance in explaining item responses compared to standalone EIRM or RF.
• Interpretable ML methods provide insights into the nonlinear and interaction effects captured by EIRM-RF.
• Empirical and simulation results confirm the advantages of the hybrid approach in modeling complex predictor relationships and random effects.

Conclusions:

• The EIRM-RF hybrid model offers a powerful approach for analyzing complex person-by-item response data.
• Combining IRTs with RF and interpretable ML enhances model accuracy and understanding of predictor effects.
• This hybrid methodology advances the field of educational measurement and psychometric modeling.