Do milestone-based rating scales improve reliability of multisource feedback?

Although studies have investigated the reliability of entrustment-based rating scales for workplace-based assessments (WBA) with one-off discrete tasks, the same has not been investigated for WBA with multiple longitudinal observations. This study fills this void by investigating the reliability of a milestone-based rating scale format for multisource feedback (MSF).

At a regional hospital providing Japan's basic postgraduate clinical training programme, an MSF instrument was developed based on a local competency framework (4 domains, 16 subdomains). First-year trainees were assessed using conventional Likert scales in Phase 1. In Phase 2, these were replaced by milestone-based rating scales. The reliability of the MSF was compared between the two phases.

Generalisability analyses showed higher Phase 2 dependability (Φ) (univariate Φ: 0.708 vs. 0.846; multivariate Φ: 0.768 vs. 0.895), indicating improved reliability. Univariate decision study results revealed that achieving Φ = 0.767 required 15 raters in Phase 1, whereas six sufficed for Φ = 0.783 in Phase 2.

Replacing the conventional Likert scale with a milestone-based scale in MSF likely contributed to the higher reliability. Future research should employ a within-subjects design to directly compare both scales.