Current Milestones Assessment Practices, Needs, and Challenges of Program Directors: A Collective Case Study in a...

Area of Science:

• Medical Education
• Graduate Medical Education Assessment
• Program Evaluation

Background:

• The Accreditation Council for Graduate Medical Education (ACGME) Milestones assessment requirement presents challenges for Program Directors (PDs) with limited assessment expertise.
• There is a need to understand how PDs and Associate PDs (APDs) effectively implement milestones assessment and evaluation in Graduate Medical Education (GME) programs.
• This study addresses the gap in understanding current practices, needs, and challenges in milestones assessment implementation within GME.

Purpose of the Study:

• To investigate the current assessment practices, needs, and challenges faced by PDs in implementing milestones assessment.
• To explore these issues within residency and fellowship programs in a pediatric hospital setting.

Main Methods:

• A collective case study approach was employed, gathering data from PDs, APDs, and Clinical Competency Committee (CCC) Chairs across 19 GME programs.
• Structured meetings with planned agendas and pre-formatted templates were used to document program needs, difficulties, and challenges.
• Cross-case thematic content analysis was conducted to identify commonalities and differences across programs.

Main Results:

• Thirty-eight participants from 19 specialties were involved. Thirteen assessment tools were commonly used, categorized as direct, indirect, and multi-source.
• Rotation evaluations (84.2%), direct observation (73.2%), and 360-degree assessments (68.4%) were primary tools for evaluating patient care competencies.
• Key challenges included logistics and tracking evaluations (52.6%), time management (47.3%), and interpreting milestones' numerical levels (31.5%). Program needs ranged from curriculum development to systematic assessment planning.

Conclusions:

• Milestones assessment and evaluation in medical education are complex and challenging but remain a priority for training programs.
• Utilizing multiple assessment methods is crucial for accurately measuring trainees' performance across essential competencies.
• The study identified significant issues PDs encounter during milestones assessment implementation, highlighting areas for improvement in GME.