Simulation-based evaluation of anaesthesia residents: optimising resource use in a competency-based assessment...

Area of Science:

• Medical Education
• Anesthesiology Training
• Simulation-Based Assessment

Background:

• Simulation training enhances anesthesiology education by providing safe, high-stakes practice.
• Implementing simulation-based assessments is resource-intensive, requiring determination of optimal scenario and faculty numbers.
• Competency-based assessment in medical education necessitates efficient and reliable evaluation methods.

Purpose of the Study:

• To determine the minimal number of simulation scenarios and faculty assessors for optimal competency-based assessment in anesthesiology.
• To examine the structure of simulation-based assessment using generalisability study methodology.
• To provide evidence for optimizing resource allocation in simulation-based medical education.

Main Methods:

• A generalisability analysis (G-analysis) was employed to examine assessment structure.
• Seventeen anesthesiology residents completed four simulations each.
• Simulations were assessed by two expert raters, with variance analyzed across scenarios, assessors, and participants.

Main Results:

• Multivariate G-analysis successfully estimated the required number of simulations and raters for optimized assessment.
• The optimal balance for accurate competency assessment was achieved with four scenarios evaluated by two simulation experts.
• Variance attributable to scenarios, assessors, and participants was quantified.

Conclusions:

• Simulation-based assessment is a vital tool for evaluating medical resident competency.
• Generalisability analysis (G-analysis) aids in planning for optimal resource utilization and cost-effectiveness in simulation-based assessments.
• This study provides a data-driven approach to designing efficient and reliable simulation-based competency evaluations.