From Quantity to Quality: A New Framework for Evaluating the Research Productivity of Residency Applicants

With USMLE Step 1 becoming pass/fail, publication count has become an increasingly important metric that medical students strive to maximize for residency applications. This has resulted in the pursuit of less meaningful research and engagement in unethical research practices (e.g., gift authorship) by students. We present a new metric designed for use by residency program directors to better evaluate a student's cumulative research contribution (CRC).

We conducted a cross-sectional study of all 2024 PGY-1 integrated United States plastic surgery residents' publication profiles. We utilized Elsevier's Scopus to identify each applicant's total number of publications and variables related to each publication including authorship rank, publication study design, and SCImago Journal Rank indicator. Two independent reviewers agreed upon study design classifications for each publication. We then developed a general CRC formula, which was used to create 48 CRC models that were used to calculate a CRC score for each applicant and compared the distribution of students' ranks using their CRC scores within each model versus total publication counts. We calculated the mean, median, standard deviation, and quartile coefficient of dispersion for the total publication count and for each CRC score distribution. We then performed Kendall's Tau-B rank correlation tests to assess the strength of association between total publication count and each CRC model, and to determine whether student rank order differed significantly across these ranking methods.

We analyzed 194 PGY-1 integrated plastic-surgery residents and 1467 publications in total.

The most frequently published study designs were retrospective cohort studies (n = 417). Mean publication count was 7.56 ± 6.77. Applicants with larger authorship teams produced more publications, but their individual contribution to the publications declined. Ranking applicants with any of the 48 CRC models significantly reshuffled student ranking relative to total publication count (τ = 0.69-0.87; all p < 0.001) and widened the spread of the cohort distribution (quartile coefficient of dispersion 0.58-0.68 for CRC models vs 0.54 for total publication count). CRC per publication showed no association with total publication count (τ ≤ 0.06, p ≥ 0.18 for all CRC models), indicating that simply publishing more does not equate to higher scholarly value. Students working in small teams (≤5 authors) achieved an average CRC score of 16.1, which was comparable to the CRC score of those in medium sized teams (5.01-10 authors) (p = 0.18). Students working in small teams (≤5 authors) achieved an average CRC score per publication of 2.52, which was significantly greater than the CRC score per publication of those in medium sized teams (5.01-10 authors) (p = 0.013). Medium sized teams (5.01-10 authors) achieved an average CRC score per publication of 2.09, which was significantly greater than the CRC score per publication of 2.18 of those in large teams (≥10 authors) (p = 0.013).

Ranking residents by publication count versus a CRC score that takes into account the quality of the contribution of a researcher's publications yields a significant difference when ranking the profiles of students who have successfully matched into integrated plastic surgery residency. Given that no metric has been publicly created for use to rank medical student research profiles, we recommend that residency programs adopt ours or a similar CRC metric when evaluating applicants. Rather than relying solely on total publication count, this approach may help encourage students to engage in higher quality research and promote ethical research practices.