Senior Surgery Residents Assessment of Junior Residents on ACGME Milestones

Residents' performance is rigorously evaluated semiannually by the ACGME-mandated clinical competency committee (CCC), using competency-based developmental outcomes (Milestones). Self-assessment (SE) using Milestones enables residents to critically evaluate their abilities. This study compares residents' milestones as assessed by 3 raters: SE, CCC, and Chief residents (CRs), to evaluate the comprehensiveness and quality of the resident review process as a potential valuable datapoint not typically included in CCC discussions.

We retrospectively analyzed Milestones scores as rated by SE, CCC, and CRs, over 5 years (July 2019-June 2024), on the 6 Core Competencies (Patient Care, Medical Knowledge, Professionalism, Interpersonal/Communication Skills, Practice-based Learning and Improvement, and Systems-based Practice). Analyses were performed using ANOVA to evaluate differences between the groups, and intraclass correlation (ICC) to assess interrater reliability between CCC-SE, SE-CR, and CCCCR assessments. All analyses were performed using SPSS.

Thirty-nine residents were evaluated at their PGY1-PGY4 levels (total 154 PGY's) by themselves, 50 faculty, and 18 chief residents (PGY5). Median (range) of evaluations per resident was 4 (2-6). Each resident has 3 evaluations completed for same Milestones by SE, CCC, and CRs. A total of 462 (154 × 3) evaluations were completed for 46 PGY1s, 43 PGY2s, 39 PGY3s, and 26 PGY4s. SE and CCC scores were similar. However, CRs score residents significantly higher than residents themselves or CCC (p < 0.001). The ICC was >0.90 between SE and CCC (p < 0.001), indicating excellent reliability, but 0.24 to 0.73 between SE and CR, and CCC and CR, indicating poor-to-moderate reliability.

Residents self-assessments closely aligned with CCC evaluations. However, CRs tended to score junior residents higher than SE and CCC. While CRs may be better positioned to comment on day-to-day work ethic, professionalism, and team dynamics, their evaluations differ from those of CCC and SE assessments. Assessing CRs' understanding of the evaluation process is necessary.