Optimal Online Calibration Designs for Item Replenishment in Adaptive Testing

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Design

Background:

• Maintaining a robust item bank is crucial for adaptive testing.
• Online calibration offers an efficient method for replenishing operational item banks with new items.

Purpose of the Study:

• To propose a novel optimal design for online item calibration, termed D-c.
• To develop Bayesian versions of optimal designs to address parameter uncertainty.
• To compare the performance of the proposed D-c design against existing methods.

Main Methods:

• Incorporating D-optimal design principles into a reformed D-optimal design (D-VR).
• Developing Bayesian optimal designs by integrating prior information for new items.
• Simulating locally optimal designs with varying calibration sample sizes and prior distribution approaches.

Main Results:

• The D-c design demonstrated superior performance, retiring fewer new items compared to the D-VR design across most examinee sample sizes.
• Bayesian D-c, utilizing priors from operational items, outperformed default priors in BILOG-MG and PARSCALE.
• Bayesian optimal designs generally surpassed locally optimal designs when initial item parameters were imprecisely estimated.

Conclusions:

• The proposed D-c design offers an efficient approach for online item calibration in adaptive testing.
• Bayesian optimal designs provide a robust strategy, particularly when dealing with uncertainty in new item parameters.
• The study highlights the benefits of incorporating prior information for enhancing the precision of online item calibration.