Using the American Board of Surgery In-Training Examination to predict board certification: a cautionary study

Area of Science:

• Surgical Education
• Medical Assessment

Background:

• The American Board of Surgery In-Training Examination (ABSITE) is intended as a formative assessment.
• However, ABSITE scores are frequently used for high-stakes decisions like resident promotion and retention.
• This study examines the predictive validity of ABSITE scores for board certification outcomes.

Purpose of the Study:

• To analyze the ability of ABSITE scores to predict success on the American Board of Surgery certification examinations.
• To evaluate the discrepancy between the intended formative use and actual high-stakes application of ABSITE scores.

Main Methods:

• Retrospective review of first-time American Board of Surgery qualifying examination (QE) and certifying examination (CE) candidates from 2006-2012.
• Linking postgraduate year (PGY) 1 and PGY5 ABSITE scores with QE and CE performance data.
• Utilizing linear and logistic regression analyses to assess predictive utility.

Main Results:

• PGY1 ABSITE scores explained 22% of QE score variance; PGY5 scores explained 30%.
• A PGY5 ABSITE threshold achieved 86% accuracy in predicting QE pass/fail, with high sensitivity (98%) but low specificity (13%).
• ABSITE scores did not demonstrate predictive ability for success on the certifying examination (CE).

Conclusions:

• ABSITE scores effectively predict qualifying examination (QE) performance but not certifying examination (CE) outcomes.
• While high ABSITE scores correlate with QE success, low scores may lead to inaccurate failure predictions.
• Program directors should incorporate diverse data sources beyond ABSITE for high-stakes resident evaluations.