Assessing Competency in Serious Illness Communication: Development of a Rater Training Program

Rigorous rater training is necessary to ensure consistent feedback. Yet, there is a lack of published recommendations for how to train raters to provide reliable, consistent assessments for communication skills, thus making competency-based training in this area challenging. We describe a method for conducting rater training for serious illness communication skills and assess interrater reliability.

We selected a previously published and validated tool for assessing serious illness communication skills. We created a rater training program adapted from a previously described program focused on team performance, making notable adjustments to tailor the program to the assessment of serious illness communication with patients given the unique challenges these conversations pose. We assessed interrater reliability at the end of the program using kappa coefficients for dichotomous checklist items and intraclass correlation coefficients for scaled items.

Five raters who are physicians with expertise in communication skills training completed the program. After training, raters assessed eight test videos. All raters achieved substantial agreement when compared to the gold standard rater for both the checklist (average overall kappa = 0.83) and scaled items (average overall intraclass correlation coefficient = 0.83).

We demonstrate an effective method for conducting rater training to assess serious illness communication skills that builds off a previously published program for team performance. Key adjustments included conducting facilitated discussions of videos and iteratively updating the rater training guide. This approach ensures reliable assessment within communication skills training.