Factor Retention in Exploratory Multidimensional Item Response Theory

Area of Science:

• Psychometrics
• Educational Measurement
• Data Science

Background:

• Multidimensional Item Response Theory (MIRT) is widely used in educational and psychological assessments.
• Accurate factor retention is critical for valid exploratory MIRT analyses.
• The comparative performance of statistical and Machine Learning (ML) methods for factor retention in MIRT is unclear.

Purpose of the Study:

• To compare the effectiveness of various statistical and ML methods for factor retention in exploratory MIRT.
• To identify the most accurate methods for determining the number of factors in MIRT analyses.

Main Methods:

• Simulated 720,000 dichotomous response datasets using MIRT under diverse conditions.
• Compared statistical methods (e.g., Kaiser Criterion, Parallel Analysis, MAP, Exploratory Graph Analysis) and ML methods (e.g., Random Forest, HistGBDT, XGBoost, ANN).
• Evaluated method performance based on correct-factoring proportions across varying data characteristics.

Main Results:

• Minimum Average Partial (MAP), Random Forest (RF), Histogram-based Gradient Boosted Decision Trees (HistGBDT), XGBoost, and Artificial Neural Network (ANN) demonstrated superior performance.
• HistGBDT generally outperformed other methods, especially when incorporating results from statistical methods as features.
• Factor retention accuracy decreased with increased data missingness and reduced sample sizes; several traditional methods showed consistent over- or under-factoring.

Conclusions:

• Machine learning methods, particularly HistGBDT, offer significant advantages for factor retention in exploratory MIRT.
• Practitioners are recommended to utilize both MAP and HistGBDT for robust factor determination in MIRT.
• Understanding the impact of data missingness and sample size is crucial for reliable MIRT analyses.