Evaluation and redesign of a system to rank applicants for surgical residencies

The present paper is a report of the evaluation and subsequent revision of a selection system for surgical house officers. The authors conducted a series of studies to determine the content validity, concurrent validity, and predictive validity of the system used by the surgery department prior to 1982. The results did not support the validity of that ranking system. In particular, the system was found not to evaluate candidates on criteria considered important by the faculty. Based on these findings, the department adopted a new ranking system organized around a multimethod-multitrait matrix (that is, using multiple methods to assess multiple traits) and based on a new set of traits that had been identified as important in the earlier validity studies. The new system has been used for five years, since 1982, with acceptance and favorable evaluation by the faculty. During that period, the median and mean rank of candidates who were matched to the residency program improved.