Evaluating communication skills in the OSCE format: reliability and generalizability

In most objective structured clinical examinations (OSCEs), communication skills are assessed as an 'add-on' to history-taking stations, rather than in stations designed to assess communication skills in the broadest sense. This study investigated the feasibility of developing such stations. In part one, 60 clinical clerks and 36 residents were rated in four 10-min emotionally charged situations portrayed by standardized patients. Inter-rater reliability was demonstrated (r = 0.59-0.63) and a highly significant effect of educational level was found. Generalizability between communication stations was low (0.17-0.20). Several explanations for poor generalizability, including poor discrimination as a result of low score variance and the confounding effect of content knowledge, were addressed in part 2. Ninety-five final-year medical students participated in an OSCE in which six 10-min encounters examined the students' ability to manage difficult emotional situations such as fear, anxiety, mania, sadness, confusion and anger. Half the students encountered a patient with moderate emotional symptoms and half an extreme emotional state. For difficult stations, students' scores were lower and standard deviation higher, suggesting that manipulating difficulty increases score variance and potentially discrimination. However, a strong interaction was found between difficulty and station content, and communication scores were highly correlated with content. Scenarios which created major communication difficulties (such as mania) resulted in much larger differences in scores between the easy and difficult versions. Communication OSCE stations can be created with acceptable reliability including difficult cases which address communication skills beyond simple history taking. Nevertheless, a generalizable set of communication skills remains elusive.