The Impact of Item Calibration Error on Variable-Length Cognitive Diagnostic Computerized Adaptive Testing.

Area of Science:

• Psychometrics
• Educational Measurement
• Computerized Adaptive Testing

Background:

• Calibration errors in item parameter estimation are a persistent challenge in test development.
• These errors can significantly distort measurement accuracy and test characteristics.

Purpose of the Study:

• To investigate the impact of calibration errors on measurement accuracy, test length, and efficiency in cognitive diagnostic computerized adaptive testing (CD-CAT).
• To examine methods for mitigating the adverse effects of calibration errors in CD-CAT.

Main Methods:

• Computer simulations were employed to assess the effects of calibration errors under different cognitive diagnostic models.
• Evaluated the performance of the deterministic input, noisy "and" gate (DINA) model, reduced reparameterized unified model, and compensatory reparameterized unified model (CRUM).

Main Results:

• Calibration errors negatively affected measurement accuracy in DINA and reduced reparameterized unified models.
• Items with calibration errors led to shorter average test lengths and overestimated test efficiency.
• The CRUM framework demonstrated greater robustness to calibration errors, maintaining stable classification accuracy, test length, and efficiency.

Conclusions:

• Calibration errors pose a significant threat to the validity of CD-CAT.
• The CRUM offers a more stable approach in the presence of calibration errors.
• Strategies including enhancing item quality, utilizing larger calibration samples, and employing cross-validation can reduce the detrimental impact of calibration errors on CD-CAT.