Development of a Reliable, Valid Procedural Checklist for Assessment of Emergency Medicine Resident Performance of...

Emergency cricothyrotomy is a rare but potentially life-saving procedure performed by emergency physicians. A comprehensive, dichotomous procedural checklist for emergency cricothyrotomy for emergency medicine (EM) resident education does not exist.

We aimed to develop a checklist containing the critical steps for performing an open emergency cricothyrotomy, to assess performance of EM residents performing an open emergency cricothyrotomy using the checklist on a simulator, and to evaluate the reliability and validity of the checklist for performing the procedure.

We developed a preliminary checklist based on literature review and sent it to experts in EM and trauma surgery. A modified Delphi approach was used to revise the checklist and reach consensus on a final version of the checklist. To assess usability of the checklist, we assessed EM residents using a cricothyrotomy task trainer. Scores were determined by the number of correctly performed items. We calculated inter-rater reliability using the Cohen kappa coefficient. Validity was assessed using the Welch t-test to compare the performance of residents who had and had not performed an open emergency cricothyrotomy, and we used analysis of variance to compare performance of postgraduate year (PGY) cohorts.

The final 27-item checklist was developed after three rounds of revisions. Inter-rater reliability was strong overall (κ = 0.812) with individual checklist items ranging from slight to nearly perfect agreement. A total of 56 residents participated, with an average score of 14.3 (52.9%). Performance varied significantly among PGY groups (P < 0.001). Residents who had performed an emergency cricothyrotomy previously performed significantly better than those who had not (P = 0.005). The developed checklist, which can be used in procedural training for open emergency cricothyrotomy, suggests that improved training approaches to teaching and assessing emergency cricothyrotomy are needed given the overall poor performance of this cohort.