Evaluating the Sampling Performance of Exploratory and Cross-Validated DETECT Procedure with Imperfect Models.

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Modeling

Background:

• Identifying the number of latent traits is crucial for multidimensional item response theory (IRT) models.
• The DETECT procedure offers a nonparametric approach to dimensionality assessment.
• Understanding DETECT's performance with real-world data is essential for applied researchers.

Purpose of the Study:

• To investigate the overall performance and outcomes of the DETECT procedure.
• To compare DETECT's results using real data with those from purely simulated data.
• To evaluate the robustness of DETECT to model misspecifications and sampling variations.

Main Methods:

• Employed a real-data sampling design and a purely simulated data set.
• Utilized a well-specified "perfect" model for data generation.
• Compared maximized DETECT values, R-ratio statistics, and item classification accuracy.

Main Results:

• The sampling behavior of DETECT and R-ratio statistics showed robustness to minor real-data factors and model misspecifications.
• Negligible differences were observed between real and simulated data sets for most outcomes.
• Item classification accuracy was nearly identical across both data types.

Conclusions:

• The DETECT procedure demonstrates robust performance in assessing dimensionality in multidimensional IRT models.
• While generally accurate, DETECT's accuracy in identifying the number of dimensions is higher with larger sample sizes in purely simulated data.
• Exploratory DETECT analysis often yielded better overall accuracy than cross-validated DETECT analysis.