Multidimensional Computerized Adaptive Testing Using Non-Compensatory Item Response Theory Models

Area of Science:

• Psychometrics
• Educational Measurement
• Computerized Adaptive Testing

Background:

• Current multidimensional computerized adaptive testing (MCAT) primarily uses compensatory multidimensional item response theory (MIRT) models.
• Challenges exist in applying MCAT with non-compensatory MIRT models due to data complexities.

Purpose of the Study:

• To develop and evaluate MCAT algorithms specifically designed for non-compensatory MIRT models.
• To compare the performance of different item selection methods within this non-compensatory framework.

Main Methods:

• Adapted and compared three item selection methods: Fisher information, mutual information, and Kullback-Leibler information.
• Conducted simulations to assess algorithm performance under varying conditions.
• Investigated the impact of termination criteria and latent trait correlations on measurement precision.

Main Results:

• Fisher information and mutual information methods demonstrated comparable and superior performance over the Kullback-Leibler method.
• Increased measurement precision and test reliability were observed with stricter termination criteria and higher latent trait correlations.
• Test reliability remained consistent across dimensions irrespective of latent trait correlations or termination criteria.

Conclusions:

• MCAT algorithms utilizing non-compensatory MIRT models can be effectively developed and implemented.
• Fisher and mutual information methods are recommended for item selection in non-compensatory MCAT.
• Findings provide insights into optimizing item bank construction for non-compensatory assessments.