Assessment of multiple physician competencies in postgraduate training: utility of the structured oral examination

Competency based medical education involves assessing physicians-in-training in multiple roles. Training programs are challenged by the need to introduce appropriate yet feasible assessment methods. We therefore examined the utility of a structured oral examination (SOE) in the assessment of the 7 CanMEDS roles (Medical Expert, Communicator, Collaborator, Manager, Health Advocate, Scholar and Professional) in a Neonatal-Perinatal Medicine subspecialty training program. Between 2004 and 2008, 68 trainees participated in an annual SOE. Each SOE consisted of 4 1st year and 4 2nd year clinical scenarios with standardized questions written by neonatologists that addressed all 7 CanMEDS roles. Examiners assigned a checklist score and global rating scores for knowledge and organization. A satisfaction survey was completed at the end of 3 examinations. Across the 5 SOEs, questions related to each competency were tabulated and an overall checklist score for each competency calculated. Inter-station reliability was determined for each CanMEDS role. Values for Cronbach's alpha were 0.62 ± 0.2 (Medical Expert), 0.43 (Communicator), 0.40 ± 0.34 (Collaborator), 0.19 ± 0.17 (Manager), 0.46 ± 0.32 (Advocate), 0.67 ± 0.18 (Scholar) and 0.79 (Professional). Inter-rater reliability, determined for the first examination when there were 2 examiners per case, was moderate to substantial for 67% of cases. Participant satisfaction was high. Electronic communication facilitated exam development and costs were minimal. The SOE demonstrates psychometric properties suitable for a formative, in-training assessment as well as low cost, ease of administration and acceptability. It may be a useful way to assess physician competencies in training programs.