Efficiency of computerized adaptive testing with a cognitively designed item bank

Area of Science:

• Psychometrics
• Educational Measurement
• Cognitive Science

Background:

• Item banks are crucial for computerized adaptive testing (CAT).
• Traditional item development is labor-intensive and expensive.
• The cognitive design system (CDS) automates item generation but uses unique difficulty calibration.

Purpose of the Study:

• To compare the measurement efficiency of CAT using item difficulty calibration versus prediction models.
• To evaluate the trade-offs between prediction accuracy and the number of items needed in CAT.

Main Methods:

• A simulation study was conducted.
• Compared a prediction model (linear logistic trait model; LLTM) against a baseline calibration model (Rasch).
• Assessed root mean square error (RMSE) and the number of additional items required for comparable accuracy.

Main Results:

• The LLTM prediction model had a higher RMSE than the Rasch model.
• However, LLTM required only a few additional items to achieve comparable RMSE.
• The number of extra items decreased as the model's explanatory rate increased.

Conclusions:

• The reduction in measurement efficiency from using prediction models is acceptable for CAT.
• Prediction models can significantly reduce or eliminate the need for costly item pretesting.
• CDS approach with prediction models offers a cost-effective alternative for item bank development.