Sequential Bayesian Ability Estimation Applied to Mixed-Format Item Tests

Area of Science:

• Educational Measurement
• Psychometrics
• Statistical Modeling

Background:

• Large-scale assessments frequently employ mixed-format items, combining multiple-choice (MC) and constructed-response (CR) formats.
• Simultaneous analysis of mixed-format items may not yield optimal ability estimates.
• Existing methods like concurrent Bayesian (CB) calibration may have limitations in accurately estimating examinee abilities.

Purpose of the Study:

• To explore a two-step sequential Bayesian (SB) analytic method for mixed item response models.
• To compare the accuracy and reliability of the SB method against a traditional concurrent Bayesian (CB) method (EAPsum).
• To evaluate the performance of the SB method across various factors including sample size and test length.

Main Methods:

• Developed and applied a two-step sequential Bayesian (SB) method integrating ability estimates from MC and CR items.
• Utilized individual-level sample-dependent prior distributions estimated from MC items for posterior ability estimation.
• Conducted simulation studies to assess parameter recovery and compared SB with CB (EAPsum) under different conditions.

Main Results:

• The SB method demonstrated more accurate and reliable ability estimation than the CB method, especially with small sample sizes (N=150, 500).
• Both methods showed comparable recovery for multiple-choice item parameters.
• The CB method slightly outperformed the SB method in recovering constructed-response item parameters, though the SB method's posterior ability estimates showed higher reliability in an empirical example.

Conclusions:

• The sequential Bayesian (SB) method provides a more accurate and reliable approach for ability estimation in mixed-format assessments compared to concurrent Bayesian (CB) methods.
• The SB method's advantages are particularly pronounced in scenarios with limited sample sizes.
• The proposed SB method offers improved posterior ability estimation reliability for mixed-item assessments.