Longitudinal Reliability of Milestones Learning Trajectories during Anesthesiology Residency

Longitudinal Milestones data reported to the Accreditation Council for Graduate Medical Education provide a structured framework for assessing the developmental progression of residents in key competencies and subcompetencies. This study aims to investigate the previously underexplored longitudinal reliability of Milestones data, with the goal of identifying patterns in learning trajectories that can inform targeted interventions for residents and programs.

A retrospective cohort study was conducted with national anesthesiology Milestones data collected from 2014 to 2020. Mixed-effects growth curve models were fit to model residents' growth trajectories. Longitudinal reliability was assessed using the indices of growth rate reliability and growth curve reliability. This study also examined variance components attributable to the factors at both the learner and program levels. Latent class growth analyses were performed to identify latent groups of learners with different learning trajectories.

The study included a total of 682,475 ratings for 4,976 learners in 140 programs. Growth curve model results indicated that the mean baseline Milestone rating across the 25 subcompetencies was 2.05 (95% CI, 1.96 to 2.14), with an average increase of 0.49 (95% CI, 0.48 to 0.51) units per reporting period. The growth rate reliability (mean ± SD, 0.58 ± 0.03) suggested a moderate capability of anesthesiology Milestones to detect individual differences in the growth of latent competency. Growth curve reliability estimates (mean ± SD, 0.71 ± 0.02) suggested acceptable overall reliability of Milestones across all the six assessment points. Significant variability was observed at both the program and learner levels ( P < 0.001). Latent class growth analyses identified 3 to 4 latent groups of learners with distinct learning trajectories across the 25 subcompetencies.

The study indicated that the anesthesiology Milestones provide moderately reliable information for tracking individual progress over time. The findings underscore the importance of using a multifaceted approach to assessment and providing individualized learning plans to support resident development.