Assessing Competency in Serious Illness Communication: Development of a Rater Training Program

Area of Science:

• Medical Education
• Communication Skills Assessment
• Healthcare Professional Training

Background:

• Consistent feedback in communication skills training requires rigorous rater training.
• Lack of published recommendations hinders reliable assessment in this critical area.
• Competency-based training for serious illness communication presents unique challenges.

Purpose of the Study:

• To describe an effective method for rater training in serious illness communication skills.
• To assess the interrater reliability of trained raters using a validated tool.

Main Methods:

• Adapted a team performance rater training program for serious illness communication.
• Utilized a previously published and validated assessment tool.
• Assessed interrater reliability using kappa coefficients and intraclass correlation coefficients.

Main Results:

• Five physician raters with communication expertise completed the training program.
• Trained raters demonstrated substantial agreement with a gold standard rater.
• Achieved high reliability scores: average kappa = 0.83 (checklist) and ICC = 0.83 (scaled items).

Conclusions:

• An effective rater training method for serious illness communication skills was demonstrated.
• Key adaptations included facilitated video discussions and iterative guide updates.
• This approach ensures reliable and consistent assessment in communication skills training.