Evaluating communication skills in the OSCE format: reliability and generalizability

Area of Science:

• Medical Education
• Clinical Skills Assessment
• Communication Skills Training

Background:

• Communication skills in Objective Structured Clinical Examinations (OSCEs) are often assessed as secondary components.
• There is a need for dedicated OSCE stations to evaluate communication skills comprehensively.

Purpose of the Study:

• To investigate the feasibility of developing OSCE stations specifically designed to assess communication skills broadly.
• To evaluate the reliability and generalizability of such communication-focused OSCE stations.

Main Methods:

• Two parts were conducted: Part 1 involved 60 clerks and 36 residents in 4 emotionally charged scenarios. Part 2 involved 95 final-year medical students in 6 encounters managing difficult emotional situations (e.g., fear, anger).
• Difficulty levels (moderate vs. extreme emotional states) were manipulated in Part 2 to assess score variance and discrimination.
• Inter-rater reliability and generalizability between stations were analyzed.

Main Results:

• Part 1 demonstrated acceptable inter-rater reliability (r = 0.59-0.63) and significant differences based on educational level.
• Generalizability between communication stations was low (0.17-0.20).
• In Part 2, difficult stations showed lower scores and higher standard deviations, increasing score variance. Communication scores correlated highly with content knowledge, particularly in challenging scenarios like mania.

Conclusions:

• Communication OSCE stations can be developed with acceptable reliability, including challenging cases.
• These stations can assess communication skills beyond basic history-taking.
• A broadly generalizable set of communication skills assessment within OSCEs remains an elusive goal.