Validity of Simulation-Based Assessment for Accreditation Council for Graduate Medical Education Milestone...

Area of Science:

• Medical Education
• Anesthesiology Training
• Simulation-Based Assessment

Background:

• Anesthesiology residents require biannual evaluations on 25 Accreditation Council for Graduate Medical Education (ACGME) milestones.
• Certain milestone domains are difficult to reliably assess during clinical practice.
• Simulation-Based Milestones Assessment (SBMA) offers a potential solution for consistent evaluation.

Purpose of the Study:

• To externally validate SBMA scores for ACGME milestones in anesthesiology residents.
• To determine if SBMA scores discriminate by postgraduate year.
• To assess if SBMA scores improve with training and correlate with traditional performance measures.

Main Methods:

• Retrospective analysis of 55 residents' SBMA data from 30 scenarios across two academic years.
• Evaluation of scenarios for postgraduate year discrimination and scoring trends over time.
• Comparison of SBMA scores with residents' clinical evaluation data.

Main Results:

• Twenty-four SBMA scenarios demonstrated discrimination based on postgraduate year.
• Significant improvements in SBMA scores were observed across training sessions (F(3, 54) = 17.79, P < 0.001).
• Moderate to strong correlations were found between SBMA scores and clinical evaluations (r values ranging from 0.43 to 0.70).

Conclusions:

• SBMA scores show significant associations with resident experience and time-in-training.
• SBMA scores correlate well with traditional clinical performance evaluations.
• SBMA is a valid metric for assessing ACGME milestone competencies in anesthesiology residents.