Evaluating Rater Accuracy in Rater-Mediated Assessments Using an Unfolding Model

Area of Science:

• Educational Measurement
• Psychometrics
• Assessment Quality

Background:

• Increasing global performance assessments necessitate improved rater quality evaluation methods.
• Current methods for assessing rater accuracy require further development.
• Ensuring reliable ratings is crucial for valid performance assessments.

Purpose of the Study:

• To introduce and evaluate an unfolding model for examining rater accuracy.
• To define accuracy based on the difference between observed and expert ratings.
• To explore the utility of the hyperbolic cosine model (HCM) in this context.

Main Methods:

• Developed an unfolding model to categorize accuracy into three latent levels: inaccurate below, accurate, and inaccurate above expert ratings.
• Utilized dichotomous accuracy ratings (0 = inaccurate, 1 = accurate).
• Applied the hyperbolic cosine model (HCM) to analyze accuracy data from a statewide writing assessment.

Main Results:

• The unfolding model successfully categorized rater accuracy into distinct latent levels.
• The hyperbolic cosine model (HCM) demonstrated effectiveness in analyzing dichotomous accuracy ratings.
• Results suggest HCM is a viable statistical approach for rater accuracy assessment.

Conclusions:

• The proposed unfolding model, utilizing HCM, is a promising method for evaluating rater accuracy.
• HCM provides a valuable interpretive framework for understanding rating quality in rater-mediated assessments.
• This approach can enhance the overall quality and validity of performance assessments.