Public Board Score Reporting Undermines Holistic Review for Residency Selection

Area of Science:

• Medical Education
• Graduate Medical Education
• Physician Training

Background:

• Holistic review is the standard for residency selection, leading many programs to de-emphasize standardized exam scores.
• Public reporting of American Board of Medical Specialties (ABMS) board pass rates may create pressure and distort applicant selection.
• Reliance on normative metrics may inadvertently increase board failure rates in programs not prioritizing test scores.

Discussion:

• The public reporting of raw board pass rates creates pressure, potentially incentivizing programs to prioritize standardized test scores over holistic review.
• Applicant cohorts differ in their baseline probability of passing certifying exams, making direct comparison of raw pass rates problematic.
• Relying solely on raw pass rates may undermine the goals of holistic review and lead to unintended consequences in residency selection.

Key Insights:

• Standardized exam scores may predict initial certifying exam success, impacting board failure rates in programs using holistic review.
• Publicly reported board pass rates can create a perverse incentive for programs to re-emphasize test scores, threatening holistic review.
• A more nuanced metric, such as an "expected board pass rate" derived from learning analytics, is needed for accurate program evaluation.

Outlook:

• ABMS member boards should consider ceasing the public reporting of raw certifying exam rates above a specific threshold for normative comparisons.
• Implementing learning analytics to establish an "expected board pass rate" could provide a more equitable and informative metric for residency program evaluation.
• This shift could better support accreditation processes and ensure residency programs focus on comprehensive applicant assessment rather than solely on test score prediction.