The Surgical Knowledge "Growth Curve": Predicting ABSITE Scores and Identifying "At-Risk" Residents

Resident performance on the American Board of Surgery In-Training Examination (ABSITE) is used for evaluation of surgical knowledge and guides resident selection for institutional remediation programs. Remediation thresholds have historically been based on ABSITE percentile scores; however, this does not account for predictors that can impact a resident's exam performance. We sought to identify predictors of yearly ABSITE performance to help identify residents "at-risk" for performing below their expected growth trajectory.

The knowledge of the residents, as measured by standardized ABSITE scores, was modeled as a function of the corresponding postgraduate year via a linear mixed effects regression model. Additional model covariates included written USMLE-1-3 examination scores, gender, number of practice questions completed, and percentage correct of practice questions. For each resident, the predicted ABSITE standard score along with a 95% bootstrap prediction interval was obtained. Both resident-specific and population-level predictions for ABSITE standard scores were also estimated.

The study was conducted at a single, large academic medical center (Massachusetts General Hospital, Boston, MA).

Six years of general surgery resident score reports at a single institution between 2014 and 2019 were deidentified and analyzed.

A total of 376 score reports from 130 residents were analyzed. Covariates that had a significant effect on the model included USMLE-1 score (PGY1: p = 0.013; PGY2: p = 0.007; PGY3: p = 0.011), USMLE-2 score (PGY1: p < 0.001; PGY2: p < 0.001; PGY3: p < 0.001; PGY4: p < 0.001; PGY5: p = 0.032), male gender (PGY1: p = 0.003; PGY2: p < 0.001; PGY3: p < 0.001; PGY4: p = 0.008), and number of practice questions completed (p=0.003). Five residents were identified as having "fallen off" their predicted knowledge curve, including a single resident on 2 occasions. Population prediction curves were obtained at 7 different covariate percentile levels (5%, 10%, 25%, 50%, 75%, 90%, and 95%) that could be used to plot predicted resident knowledge progress.

Performance on USMLE-1 and -2 examinations, male gender, and number of practice questions completed were positive predictors of ABSITE performance. Creating residency-wide knowledge growth curves as well as individualized predictive ABSITE performance models allows for more efficient identification of residents potentially at risk for poor ABSITE performance and structured monitoring of surgical knowledge progression.