Do milestone-based rating scales improve reliability of multisource feedback?

Area of Science:

• Medical Education
• Assessment Reliability
• Postgraduate Training

Background:

• Workplace-based assessments (WBA) using entrustment-based scales are studied for discrete tasks, but reliability for longitudinal observations is less explored.
• Multisource feedback (MSF) is crucial for evaluating trainees in postgraduate medical education.

Purpose of the Study:

• To investigate the reliability of a milestone-based rating scale format for MSF in longitudinal WBA.
• To compare the reliability of milestone-based scales versus conventional Likert scales in MSF.

Main Methods:

• Developed an MSF instrument based on a local competency framework for first-year postgraduate trainees.
• Implemented conventional Likert scales in Phase 1 and milestone-based rating scales in Phase 2.
• Utilized generalisability and decision study analyses to compare MSF reliability between phases.

Main Results:

• Generalisability analyses showed significantly higher dependability (Φ) in Phase 2 (milestone-based) compared to Phase 1 (Likert scale).
• Univariate dependability improved from 0.708 to 0.846, and multivariate dependability from 0.768 to 0.895.
• Achieving a target dependability required fewer raters with the milestone-based scale (6 raters) versus the Likert scale (15 raters).

Conclusions:

• Replacing Likert scales with milestone-based scales in MSF likely enhances assessment reliability.
• Milestone-based scales may increase assessment efficiency by requiring fewer raters.
• Further research using within-subjects designs is recommended to directly compare scale effectiveness.