A Checklist to Help Faculty Assess ACGME Milestones in a Video-Recorded OSCE.

Area of Science:

• Medical Education
• Anesthesiology Training
• Performance Assessment

Background:

• Accreditation Council for Graduate Medical Education (ACGME) Milestones are essential for assessing resident performance.
• Faculty face challenges in accurately evaluating resident competency using current assessment tools.

Purpose of the Study:

• To evaluate if a checklist improves the quality of faculty milestone assessments during an objective structured clinical examination (OSCE).
• The study focused on OSCEs simulating adverse event disclosure scenarios.

Main Methods:

• A randomized study involving 20 anesthesiology faculty members from 3 institutions.
• Faculty assessed trainee performance in 5 OSCE videos, with one group using milestones alone and the other using milestones plus a 13-item checklist.
• Logistic regression analyzed the impact of checklist use on correct performance classification.

Main Results:

• Most faculty (13/20) found milestone-only assessment difficult.
• Checklist use significantly increased correct classification odds for entry-level (OR=9.2) and junior-level (OR=2.7) performance.
• Checklist use did not significantly impact assessments at higher competency levels.

Conclusions:

• Many anesthesiology faculty struggle to assess videotaped OSCE performance of error disclosure using milestones.
• A checklist significantly aids faculty in correctly assessing resident performance at entry and junior competency levels.