Predicting academic performance in surgical training

During surgical residency, trainees are expected to master all the 6 competencies specified by the ACGME. Surgical training programs are also evaluated, in part, by the residency review committee based on the percentage of graduates of the program who successfully complete the qualifying examination and the certification examination of the American Board of Surgery in the first attempt. Many program directors (PDs) use the American Board of Surgery In-Training Examination (ABSITE) as an indicator of future performance on the qualifying examination. Failure to meet an individual program's standard may result in remediation or a delay in promotion to the next level of training. Remediation is expensive in terms of not only dollars but also resources, faculty time, and potential program disruptions. We embarked on an exploratory study to determine if residents who might be at risk for substandard performance on the ABSITE could be identified based on the individual resident's behavior and motivational characteristics. If such were possible, then PDs would have the opportunity to be proactive in developing a curriculum tailored to an individual resident, providing a greater opportunity for success in meeting the program's standards.

Overall, 7 surgical training programs agreed to participate in this initial study and residents were recruited to voluntarily participate. Each participant completed an online assessment that characterizes an individual's behavioral style, motivators, and Acumen Index. Residents completed the assessment using a code name assigned by each individual PD or their designee. Assessments and the residents' 2013 ABSITE scores were forwarded for analysis using only the code name, thus insuring anonymity. Residents were grouped into those who took the junior examination, senior examination, and pass/fail categories. A passing score of ≥70% correct was chosen a priori. Correlations were performed using logistic regression and data were also entered into a neural network (NN) to develop a model that would explain performance based on data obtained from the TriMetrix assessments.

A total of 117 residents' TriMetrix and ABSITE scores were available for analysis. They were divided into 2 groups of 64 senior residents and 53 junior residents. For each group, the pass/fail criteria for the ABSITE were set at 70 and greater as passing and 69 and lower as failing. Multiple logistic regression analysis was complete for pass/fail vs the TriMetrix assessments. For the senior data group, it was found that the parameter Theoretical correlates with pass rate (p < 0.043, B = -0.513, exp(B) = 0.599), which means increasing theoretical scores yields a decreasing likelihood of passing in the examination. For the junior data, the parameter Internal Role Awareness correlated with pass/fail rate (p < 0.004, B = 0.66, exp(B) = 1.935), which means that an increasing Internal Role Awareness score increases the likelihood of a passing score. The NN was able to be trained to predict ABSITE performance with surprising accuracy for both junior and senior residents.

Behavioral, motivational, and acumen characteristics can be useful to identify residents "at risk" for substandard performance on the ABSITE. Armed with this information, PDs have the opportunity to intervene proactively to offer these residents a greater chance for success. The NN was capable of developing a model that explained performance on the examination for both the junior and the senior examinations. Subsequent testing is needed to determine if the NN is a good predictive tool for performance on this examination.