Reading the Writing on the Wall: Resident Evaluation Measures as Predictors of Performance in Practice

Identifying orthopaedic surgery residents at risk of future professional or practice-related difficulties remains challenging. While knowledge-based assessments such as the Orthopaedic In-Training Examination (OITE) are widely used, they incompletely capture behavioral and professionalism concerns that often underlie significant problems during residency and subsequent practice.

We conducted a retrospective, single-institution cohort study of 119 orthopaedic surgery residents who entered training between 1978 and 2013. Institutional resident files were reviewed for documented performance problems in academic, clinical, and professionalism domains, OITE scores, and American Board of Orthopaedic Surgery (ABOS) Part I and Part II examination outcomes. Adverse postresidency indicators were assessed using publicly available data, including board examination failure, licensure status, practice continuity, and persistent negative online presence. Receiver operating characteristic (ROC) curve analysis was used to evaluate the predictive utility of cumulative documented performance problems and OITE scores.

Forty-two residents (35%) had at least 1 documented performance problem during residency (mean 4.98, range 1-13). Residents with documented problems were significantly more likely to fail ABOS Part I (33% vs. 8%; relative risk [RR] 4.28, p = 0.002) and Part II (24% vs. 4%; RR 6.11, p = 0.004) and to demonstrate persistent negative online presence (19% vs. 5%; RR 3.67, p = 0.026). Having more than 1 adverse postresidency indicator was also more common among residents with documented problems (31% vs. 6%; RR 4.77, p = 0.001). On ROC analysis, 4 or more documented problems predicted failure on ABOS Part I (AUC 0.787) and the presence of adverse postresidency indicators (AUC 0.772), while 3 or more problems best predicted ABOS Part II failure (AUC 0.784). OITE scores demonstrated lower predictive accuracy (AUC 0.696) and lacked a clearly actionable threshold.

Cumulative documented performance problems during residency-particularly behavioral and professionalism-related concerns-are associated with increased risk of board examination failure and adverse practice outcomes after graduation. Aggregated longitudinal documentation may provide an early, complementary signal of risk beyond knowledge-based testing.