A procedure for dimensionality analyses of response data from various test designs

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Analysis

Background:

• Traditional dimensionality analyses face challenges with test designs where many item pairs are not administered.
• Computerized adaptive testing (CAT) and multistage testing (MST) present unique data structures complicating item analysis.

Purpose of the Study:

• To propose a modified DETECT index for dimensionality analysis in sparse data from complex testing designs.
• To evaluate the reliability and accuracy of the modified DETECT index in identifying multidimensional structures.

Main Methods:

• Development and theoretical proof of a modified DETECT index.
• Decomposition of the modified DETECT index into reliability indices.
• Simulation studies to assess performance under various conditions.
• Application to real-world data from two-stage tests.

Main Results:

• The modified DETECT index successfully partitions items based on dimensionality under specific conditions.
• Simulation results demonstrate the index's ability to recover the true dimensional structure of response data.
• The decomposed parts of the index provide measures of reliability for multidimensional data analysis.

Conclusions:

• The modified DETECT procedure offers a robust solution for dimensionality analysis in sparse test data.
• The proposed method enhances the reliability and interpretability of dimensionality assessments in CAT and MST.
• Practical application to real data confirms the utility of the modified DETECT index.