Variability in Learner Performance Using the ACGME Harmonized Milestones During the First Year of Postgraduate...

Postgraduate medical training in the United States requires formative assessments of learners using the Accreditation Council for Graduate Medical Education (ACGME) milestones system. With Milestones 2.0, Harmonized Milestones (HMs) for 4 competency domains (professionalism, communication and interpersonal skills, systems-based practice, and practice-based learning and improvement) across specialties were developed. Performance of postgraduate trainees across specialties and at the transition to residency can be explored with the HMs. This study examined the factors that contribute to the variability in the assessments of postgraduate year 1 (PGY-1) learners as measured using Milestones 2.0.

This retrospective study assessed national ACGME HM data from PGY-1 residents at U.S. medical schools in July 2021 and 2022 from the 6 largest specialties: emergency medicine, family medicine, internal medicine, general surgery, psychiatry, and pediatrics. Variance component analyses were conducted using cross-classified random-effects models, accounting for clustering; estimated variance components were used to generate inference on contribution of learner variability due to residency program, medical school, and specialty and make inferences on HM rating practices, including straight-lining.

The sample included 57,132 PGY-1 residents (2,430 programs). Specialty accounted for the largest variance (22%) across HM competency domains. Within specialty, variance components for trainees, residency programs, and medical schools accounted for 22%, 35%, and 2% of total variance, respectively. Straight-lining was found at 6 months for 6,827 of 56,804 PGY-1 residents (12%), with the greatest amount in surgery (2,105 of 5,559 [38%]).

This study found variability in HM performance across 6 specialties due to medical schools, specialty, residency programs, and trainees with limited variability attributed to medical school and learner. Substantial differences across specialties call for the need for clinical educators, researchers, and accreditors to create a shared mental model to bolster the evaluative strength of milestones and prepare residents for the needs of health care.