Faculty-Resident Concordance in Workplace-Based Operative Assessment Using the Society for Improving Medical...

To measure faculty-trainee agreement regarding operative autonomy and performance and identify facilitators and barriers to engagement.

Urology faculty and trainees completed the Society for Improving Medical Professional Learning assessments within 72 hours of operative cases performed between August 2019 and July 2020. Trainee autonomy, performance, and case complexity were measured. Faculty-trainee agreement was quantified using weighted kappa coefficients and 95% confidence intervals. Post-intervention participant reactions were elicited via survey for feasibility assessment.

Four hundred eighty assessments were completed (17 trainees, 52.1%; 20 faculty, 47.9%). 188 paired assessments were analyzed. Faculty-trainee agreement was moderate for autonomy (kappa = 0.55, 95% CI: 0.45-0.65) and surgical performance (kappa = 0.47, 95% CI: 0.34-0.59), but low for case complexity (kappa = 0.38, 95% CI: 0.27-0.50). The most common criticism was lack of reminders for faculty to complete assessments, cited by 50% of residents and 37% of faculty.

Our institution is the first to integrate Society for Improving Medical Professional Learning assessment into urology residency training, demonstrating its feasibility for operative assessment. Moderate agreement was exhibited in autonomy and performance assessments, while lower agreement on case complexity suggests differing perceptions of surgical difficulty. Notably, faculty rated trainees higher in autonomy compared to self-assessments. Future investigations should explore contributing factors to these differences and address barriers, such as reminders for assessment completion.