Building the Building Blocks: Development of a Mission-Driven Residency Application Scoring Rubric for Holistic...

Traditional residency application review methods are fraught with bias. Literature on how to create and use rubrics to aid in holistic review is lacking. The authors describe their residency program's process of designing and implementing a mission-based rubric to aid in holistic review.

After updating the University of Pittsburgh Medical Center internal medicine-pediatrics residency program's mission statement in April 2020, the authors used experiences, attributes, and metrics domains to create a rubric based on the Electronic Residency Application Service application. The rubric included sections on the personal statement, Medical Student Performance Evaluation notable points and curriculum vitae data, additional positives, academic metrics, and red flags, with each section weighted based on alignment with the program's mission and values.

The authors, as program leadership, found the rubric to be clear, efficient, and replicable, requiring ~15-20 minutes to read, review, and score each application. The rubric aided in equitable scoring across reviewers, with underrepresented in medicine (URM) applicants scoring similarly to non-URM applicants for the 2020-2021, 2021-2022, and 2022-2023 seasons, and led to reproducible results with an intraclass correlation coefficient of 0.82. The rubric led to an increase in the number of applicants who identify as URM in the initial top 10 rank list each season, as compared to the prior 3 seasons, and the holistic review scores served as a starting point for a final rank list.

Larger programs could scale similar holistic review processes by training additional reviewers to use the rubric or implementing it at various stages in the recruitment process. The authors were prompted to consider and improve ways their program could support potential residents. By implementing holistic review, programs can recruit diverse applicant classes that reflect their mission, values, and communities served, while mitigating biases inherent to graduate medical education systems.