Clinical performance-based test sensitivity and specificity in predicting first-year residency performance

To assess the use of a standardized-patient-based postclerkship examination (PCX) to predict students' performances in their first year of residency, the authors used data from 202 students of the classes of 1987, 1988, and 1990 at the Southern Illinois University School of Medicine. The PCX was found to be more sensitive than specific in that it identified more correctly those students who received high ratings in their first year of residency than those who received low ratings. Consequently, while the rate of false negatives was low across the three classes, the rate of false positives was relatively high. Analyses of the supervisors' written comments on residents' performances indicated that more than half of the false positives and false negatives resulted mostly from problems found with the supervisors' ratings, rather than from the inaccuracy of the PCX in predicting students' residency performances. The supervisors' ratings did not always match their written comments and did not always represent pure assessments of the residents' cognitive performances. Finally, because of the large number of missing residency performance ratings among the graduates who had performed unsatisfactorily on the PCX, it is expected that the sensitivity of the PCX may be overestimated and its specificity underestimated. Overall, the results suggest that the standardized-patient-based PCX is a useful indicator of students' readiness for and performance in residency.