Modeling Rating Order Effects Under Item Response Theory Models for Rater-Mediated Assessments.

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Modeling

Background:

• Rater effects are prevalent in assessments where human judgment is involved.
• Existing item response theory (IRT) models primarily address static rater effects.
• Dynamic rater effects, such as judgment fatigue, can impact rating quality over time.

Purpose of the Study:

• To develop new many-faceted (MF)-IRT models accounting for dynamic rater effects, specifically the rating order effect.
• To investigate the impact of rating order on ratee scores and rater severity.
• To provide a framework for analyzing rater-mediated assessments where rater performance may change.

Main Methods:

• Development of two MF-IRT models incorporating systematic or stochastic rater severity drift.
• Utilizing Bayesian estimation for parameter estimation in the proposed models.
• Conducting two simulation studies to evaluate model performance and parameter recovery.

Main Results:

• The newly developed MF-IRT models demonstrated satisfactory parameter estimation via Bayesian methods.
• Disregarding the rating order effect led to biased estimations of model structure and ratee proficiency.
• Simulations confirmed the necessity of accounting for dynamic rater effects in IRT.

Conclusions:

• The proposed MF-IRT models effectively capture dynamic rater effects like the rating order effect.
• Failure to account for rating order bias can significantly compromise the validity of assessment results.
• The models offer a robust approach for analyzing rater-mediated evaluations, improving accuracy and fairness.