The Effects of Rating Designs on Rater Classification Accuracy and Rater Measurement Precision in Large-Scale...

Area of Science:

• Educational measurement
• Psychometrics
• Assessment design

Background:

• Existing research on rating designs focuses on rater effects and student achievement.
• Limited guidance exists on how rating designs impact rater accuracy and precision.
• Performance assessments and mixed-format assessments require understanding rater behavior.

Purpose of the Study:

• To investigate the impact of different rating designs on rater classification accuracy and measurement precision.
• To compare rating designs within the context of mixed-format assessments.
• To provide practical recommendations for assessment design.

Main Methods:

• Analysis of National Assessment of Educational Progress (NAEP) data.
• Simulation studies to systematically explore rating design impacts.
• Evaluation of rater classification accuracy (severe/lenient) and measurement precision.

Main Results:

• The complete rating design demonstrated the highest rater classification accuracy and measurement precision.
• The multiple-choice (MC) + spiral link design and MC link design followed in performance.
• The MC + spiral link design offers a practical balance between cost and effectiveness.

Conclusions:

• Rating design significantly influences rater classification accuracy and measurement precision in mixed-format assessments.
• While complete rating designs are optimal, MC + spiral link designs present a viable alternative.
• Findings inform best practices for designing assessments and interpreting rater data.