Reliability and validity of a tool to assess airway management skills in anesthesia trainees

Gaining expertise in procedural skills is essential for achieving clinical competence during anesthesia training. Supervisors have the important responsibility of deciding when the trainee can be allowed to perform various procedures without direct supervision while ensuring patient safety. This requires robust and reliable assessment techniques. Airway management with bag-mask ventilation and tracheal intubation are routinely performed by anesthesia trainees at induction of anesthesia and to save lives during a cardiorespiratory arrest. The purpose of this study was to evaluate the construct validity, and inter-rater and test-retest reliability of a tool designed to assess competence in bag-mask ventilation followed by tracheal intubation in anesthesia trainees.

Informed consent was obtained from all participants. Tracheal intubation and bag-mask ventilation skills in 10 junior and 10 senior anesthesia trainees were assessed by two investigators on two occasions at a 3-4 weeks interval, using a procedure-specific assessment tool.

Average kappa value for inter-rater reliability was 0.91 and 0.99 for the first and second assessments, respectively, with an average agreement of 95%. The average agreement for test-retest reliability was 82% with a kappa value of 0.39. Senior trainees obtained higher scores compared to junior trainees in all areas of assessment, with a significant difference for patient positioning, preoxygenation, and laryngoscopy technique, depicting good construct validity.

The tool designed to assess bag-mask ventilation and tracheal intubation skills in anesthesia trainees demonstrated excellent inter-rater reliability, fair test-retest reliability, and good construct validity. The authors recommend its use for formative and summative assessment of junior anesthesia trainees.