Using Precept-Assist® to predict performance on the American Board of Family Medicine In-Training Examination

Area of Science:

• Medical Education
• Family Medicine Residency Training
• Assessment Tools

Background:

• Precept-Assist® (PA) is a computer-based program used in family medicine residency to evaluate resident knowledge.
• Attending physicians score residents on a Likert-type scale for each precept.
• Validation of the PA program for family medicine residents was the primary objective.

Purpose of the Study:

• To validate the Precept-Assist® (PA) program as an assessment tool for family medicine residents.
• To determine the correlation between PA scores and standardized in-training exam performance.
• To assess the predictive value of PA scores for future resident performance.

Main Methods:

• A validation study design was employed.
• Data included PA and American Board of Family Medicine (ABFM) In-Training Exam (ITE) scores from 216 residents (2002-2011).
• Pearson correlation coefficients and Receiver Operating Characteristic (ROC) curve analysis were used to analyze the data.

Main Results:

• A moderately positive correlation (r=0.55) was found between PA scores and ABFM ITE scores.
• The Area Under the Curve (AUC) for the ROC curve was 0.783.
• A PA score of 5.5 demonstrated 72% sensitivity and 77% specificity in predicting ITE scores of 500 or above.

Conclusions:

• PA scores are significantly correlated with ABFM In-Training Exam scores.
• The Precept-Assist® program is a valid screening tool.
• PA can predict future performance on Family Medicine In-Training exams.