Making robust assessments of specialist trainees' workplace performance.

Area of Science:

• Medical Education
• Anaesthesiology Training
• Workplace-Based Assessment

Background:

• Workplace-based assessments are crucial for trainee performance evaluation but have shown inconsistent success.
• The mini-clinical evaluation exercise (mini-CEX) is a compulsory assessment in anaesthesia training.
• Implementing an entrustability scale for mini-CEX could enhance its utility and reliability.

Purpose of the Study:

• To evaluate the reliability and validity of mini-CEX assessments using an entrustability scale in a large anaesthesia training program.
• To determine if adjusting supervisor scores for expected supervision requirements improves assessment reliability.
• To assess the feasibility of using mini-CEX for high-stakes decisions in trainee progression.

Main Methods:

• Analysis of 7808 mini-CEX scores from Australian and New Zealand anaesthesia trainees over 12 months.
• Adjustment of supervisor scores based on the expected supervision level for trainees at different training stages.
• Application of generalisability theory to assess score reliability.

Main Results:

• Supervision requirements decreased with increased training duration and level, supporting assessment validity.
• Moderate reliability (G > 0.7) was achieved with a feasible number of assessments.
• Adjusting scores significantly improved reliability (G > 0.8 with nine assessments), and identified 3% of trainees below standard.

Conclusions:

• The entrustability-based mini-CEX demonstrates moderate reliability and validity in anaesthesia training.
• Score adjustment enhances reliability and aids in identifying underperforming trainees.
• The mini-CEX, when used with this scoring system, is reliable enough for high-stakes decisions regarding trainee progression.