An Iterative Parametric Bootstrap Approach to Evaluating Rater Fit.

Area of Science:

• Educational measurement
• Psychometrics
• Item Response Theory (IRT)

Background:

• Performance assessments utilize measurement theory models to identify rater discrepancies.
• Infit and outfit mean square error (MSE) statistics are commonly used to detect problematic scoring patterns.
• Traditional rule-of-thumb critical values for MSE statistics may not be appropriate in many empirical settings.

Purpose of the Study:

• To evaluate the effectiveness of parametric bootstrapped critical values for detecting rater misfit.
• To assess the false-positive and true-positive rates of infit and outfit MSE statistics using a simulation study.
• To propose and evaluate an iterative parametric bootstrap procedure to improve rater misfit detection.

Main Methods:

• A simulation study was conducted to assess the performance of traditional parametric bootstrap and rule-of-thumb critical values.
• An iterative parametric bootstrap procedure was developed and implemented.
• False-positive and true-positive rates of infit and outfit MSE statistics were analyzed.

Main Results:

• Traditional parametric bootstrap and rule-of-thumb methods exhibited inflated false-positive rates and low true-positive rates for rater misfit detection.
• The proposed iterative parametric bootstrap procedure demonstrated better control over false-positive rates.
• The iterative procedure also yielded higher true-positive rates compared to traditional methods.

Conclusions:

• Standard methods for interpreting infit and outfit MSE statistics in performance assessments have limitations.
• An iterative parametric bootstrap procedure provides a more accurate and reliable approach for identifying rater misfit.
• This improved method enhances the validity of performance assessment evaluations.