Evaluation of assessment methods of cardiac arrest simulations

Evaluation of advanced cardiac life support performance requires a consistent assessment process. This study compares a new objective method of evaluation to the currently used method to determine if this objective method improves evaluator concordance on critical errors and whether or not this method is reasonable when compared to the current evaluation method. The design compared two experienced evaluator groups consisting of two evaluators in each group, with one group utilizing written guidelines and the other without these guidelines. Each evaluator independently reviewed student performances on 36 videotaped simulations. Results indicate that evaluators with the written guidelines identified critical errors more often than evaluators without these guidelines (30 versus 20 of 36 cases). When the critical errors observed by the no guideline evaluators were compared to the written guidelines, the critical errors were consistent with the written guidelines in 32 of 36 cases. These results suggest that written guidelines improve evaluator identification of critical errors observed and are reasonable when compared to two experienced evaluators without these guidelines.