Poor inter-rater reliability on mock anesthesia oral examinations

Area of Science:

• Medical Education
• Anesthesiology Assessment

Background:

• Inter-rater reliability (IRR) and communication skills significantly impact oral examination scores.
• Assessing the consistency of grading and the effectiveness of training is crucial for standardized medical evaluations.

Purpose of the Study:

• To evaluate the IRR among American Board of Anesthesiology (ABA) and Royal College of Physicians and Surgeons of Canada (RCPSC) graders working in isolation.
• To determine the impact of specific training in examination techniques on resident performance.

Main Methods:

• A randomized, pretest-posttest trial involving 25 residents who completed two oral examinations (E1 and E2).
• Residents were randomized into a routine education group or an intervention group receiving communication skills training.
• Videotaped examinations were scored by experienced ABA and RCPSC graders.

Main Results:

• Initial IRR was very poor (weighted Kappa = 0.166), with slight improvement in E2 (weighted Kappa = 0.275).
• The pass rate for grader pairs significantly increased from E1 (15%) to E2 (43%), benefiting both groups.
• While specific communication skills training did not show a significant effect, practice orals appeared to enhance future examination performance.

Conclusions:

• Graders scoring oral examinations in isolation may exhibit poor inter-rater reliability.
• Teaching communication and presentation techniques did not demonstrably improve resident performance in this study.
• Practice oral examinations may be beneficial for preparing residents for future assessments.