The OSSE: Development and validation of an "Objective Structured Supervision Evaluation"

Trainee supervision and teaching are distinct skills that both require faculty physician competence to ensure patient safety. No standard approach exists to teach physician supervisory competence, resulting in variable trainee oversight and safety threats. The Objective Structured Teaching Evaluation (OSTE) does not adequately incorporate the specific skills required for effective supervision. To address this continuing medical education gap, the authors aimed to develop and identify validity evidence for an "Objective Structured Supervision Evaluation" (OSSE) for attending physicians, conceptually modeled on the historic OSTE.

An expert panel used an iterative process to create an OSSE instrument, which was a checklist of key supervision items to be evaluated during a simulated endotracheal intubation scenario. Three trained "standardized residents" scored faculty participants' performance using the instrument. Validity testing modeled a contemporary approach using Kane's framework. Participants underwent simulation-based mastery learning (SBML) with deliberate practice until meeting a minimum passing standard (MPS).

The final instrument contained 19 items, including three global rating measures. Testing domains included supervision climate, participant control of patient care, trainee evaluation, instructional skills, case-specific measures, and overall supervisor rating. Reliability of the assessment tool was excellent (ICC range 0.84-0.89). The assessment tool had good internal consistency (Cronbach's α = 0.813). Out of 24 faculty participants, 17 (70.8%) met the MPS on initial assessment. All met the MPS after SBML and average score increased by 19.5% (95% CI of the difference 10.3%-28.8%, p = 0.002).