Do You Really Mean It? Assessing the Strength, Frequency, and Reliability of Applicant Commitment Statements During...

Despite the critical nature of the residency interview process, few metrics have been shown to adequately predict applicant success in matching to a given program. While evaluating and ranking potential candidates, bias can occur when applicants make commitment statements to a program. Survey data show that pressure to demonstrate commitment leads applicants to express commitment to multiple institutions including telling >1 program that they will rank them #1. The primary purpose of this cross-sectional observational study is to evaluate the frequency of commitment statements from applicants to 5 anesthesiology departments during a single interview season, report how often each statement is associated with a successful match, and identify how frequently candidates incorrectly represented commitments to rank a program #1.

During the 2014 interview season, 5 participating anesthesiology programs collected written and verbal communications from applicants. Three residency program directors independently reviewed the statements to classify them into 1 of 3 categories; guaranteed commitment, high rank commitment, or strong interest. Each institution provided a deidentified rank list with associated commitment statements, biographical data, whether candidates were ranked-to-match, and if they successfully matched.

Program directors consistently differentiated among strong interest, high rank, and guaranteed commitment statements with κ coefficients of 0.9 (95% CI, 0.8-0.9) or greater between any pair of reviewers. Overall, 35.8% of applicants (226/632) provided a statement demonstrating at least strong interest and 5.4% (34/632) gave guaranteed commitment statements. Guaranteed commitment statements resulted in a 95.7% match rate to that program in comparison to statements of high rank (25.6%), strong interest (14.6%), and those who provided no statement (5.9%). For those providing guaranteed commitment statements, it can be assumed that the 1 candidate (4.3%) who did not match incorrectly represented himself. Variables such as couples match, "R" positions, and not being ranked-to-match on both advanced and categorical rank lists were eliminated because they can result in a nonmatch despite truthfully ranking a program #1.

Each level of commitment statement resulted in a progressively increased frequency of a successful match to the recipient program. Only 5.4% of applicants committed to rank a program #1, but these statements were very reliable. These data can help program directors interpret commitment statements and assist accurate evaluation of the interest of candidates throughout the match process.