Variability in Learner Performance Using the ACGME Harmonized Milestones During the First Year of Postgraduate...

Area of Science:

• Medical Education
• Graduate Medical Training
• Competency-Based Assessment

Background:

• The Accreditation Council for Graduate Medical Education (ACGME) Milestones system is crucial for formative assessments in US postgraduate medical training.
• Milestones 2.0 introduced Harmonized Milestones (HMs) across specialties to standardize competency domain evaluation.
• Understanding variability in these assessments is key for improving training and resident preparedness.

Purpose of the Study:

• To examine factors contributing to assessment variability in postgraduate year 1 (PGY-1) learners using Milestones 2.0.
• To analyze the sources of variance in Harmonized Milestones (HMs) performance across different specialties and training levels.

Main Methods:

• Retrospective analysis of national ACGME HM data for PGY-1 residents in 6 major US specialties (July 2021-2022).
• Variance component analyses using cross-classified random-effects models to assess contributions from learner, program, medical school, and specialty.
• Investigation of rating practices, including "straight-lining" (consistent ratings).

Main Results:

• Specialty accounted for the largest variance (22%) in HM competency domains.
• Within specialties, trainees (22%), residency programs (35%), and medical schools (2%) contributed to variance.
• Straight-lining occurred in 12% of residents, most notably in surgery (38%).

Conclusions:

• Significant variability in HM performance exists across specialties, influenced by programs and trainees, with less impact from medical schools.
• Disparities across specialties necessitate a shared mental model among educators, researchers, and accreditors.
• Enhancing the evaluative strength of milestones is vital for preparing residents for healthcare demands.