Comparing the effects of small, large, and two-tier signals on residency interview invitations

Beginning in the 2023-2024 application cycle, specialties began to try out different approaches to program signaling. This study explores the effects of program signals in deciding whom to interview across small number signal (SNS), large number signal (LNS) and two-tier signal (2TS) approaches.

Of the 22 specialties that used signaling in the Electronic Residency Application Service (ERAS) in the 2023-2024 cycle, dermatology, emergency medicine, general surgery, neurological surgery, obstetrics and gynecology, and orthopaedic surgery were analyzed as representative of the signaling approaches being studied. Data on signals, applicant type, passing USMLE Step 1, USMLE Step 2 CK score, school and program state were collected from the MyERAS application; interview data was collected from ERAS PDWS and/or Thalamus. Multilevel logistic regression was used to assess the relationship between signals, other application data, and interview invitation.

The presence of a signal increased interview likelihood regardless of approach. Applicants who signaled in SNS specialties observed a 4.7 to 7.4 times the odds of receiving interview offers compared to 8.4 to 18.5 times the odds in LNS, and up to 55 to 218 times the odds for gold signals in the 2TS approach. In-state status, applicant type, never failing Step 1, and Step 2 CK scores contributed to interview selection, though the effects were not as great as the signal contribution. The effect of signaling was stronger for out-of-state applicants.

The authors report a strong association between signaling and the likelihood of an interview regardless of signaling method, though the effect was greatest with two-tier and large signal approaches. There was variation in the use of signals and the weight a program allocates to signals. Never having failed the Step 1 exam, state alignment, applicant type, and Step 2 CK scores were also associated with interview invitation.