A Structured Approach to Integrating Holistic Review Into Rank Order List Construction

Residency programs are increasingly seeking to incorporate holistic review into the selection of applicants to interview, but its integration into the creation of the rank order list (ROL) remains less known.

We sought to develop a formula that reflects holistic review of an applicant by incorporating key application elements identified by the residency program and that aligns with the final submitted ROL.

For the 2023-2024 application cycle, the leadership team from a single emergency medicine residency training program identified key attributes that characterized a successful resident. We then created a formula that weighted the identified elements to reflect our standard approach to applicant assessment. After submission of our ROL to the National Resident Matching Program, we compared the formula's performance to that of the final submitted ROL using linear regression.

A total of 143 of 160 interviewed applicants placed on the final ROL and were analyzed. Seventeen applicants that were unranked or missing a Standardized Letter of Evaluation were excluded. The regression analysis showed that the rank formula closely approximated the final position on the ROL (β = -14.38, 95% CI 15.94 to -12.83; R ² = 0.69; p < 0.0001). A higher rank score was strongly associated with a lower ROL number, which indicated a more favorable placement.

Using a ROL formula may consistently sort applicants post-interview and allow a leadership team to see when an applicant has moved outside their projected range, ensuring standardized placement on the ROL based on predefined attributes. We demonstrated a model that may assist in the comprehensive ranking of our residency applicants by incorporating elements of holistic review.