Consensus evidence evaluation in resuscitation research: analysis of Type I and Type II errors

This paper addresses the following statistical question: 'if genuine improvements in cardiopulmonary resuscitation (CPR) were discovered that doubled the probability of resuscitation success in a series of randomized clinical trials, would they be recognized and incorporated into consensus guidelines?'.

Statistical powers for hypothetical individual clinical trials comparing experimental and control CPR were computed as a function of the study N when the true probabilities for immediate survival, 24 h survival, and discharge survival in the experimental group were twice those in the control group. Next, the binomial distributions describing the numbers of statistically significant studies in a series of equally powered trials of the same intervention were determined. These were compared with varying criteria for consensus among expert reviewers, expressed in terms of the number of 'positive' studies showing a statistically significant difference that reviewers would require before approving the experimental method.

False-negative evaluations (i.e. failures to approve a technique that actually doubled survival) were extremely common under a wide range of realistic assumptions and consensus criteria, especially when simulated long-term survival data were considered. Similar methods showed that false-positive evaluations would be extremely rare, provided that at least two of the clinical trials in a series showed a statistically significant benefit of the experimental method.

Optimization of evidence evaluation can and should be carried out to make better use of available data in creating resuscitation guidelines. One simple approach is the 'two and one quarter test': if at least two well-conducted studies in a series are significantly positive (P<0.05) comprising at least one-quarter of all studies in the series, a positive effect can be inferred with small Type I and Type II errors. In addition, greater reliance on modern, unbiased methods such as cumulative meta-analysis is needed to increase the sensitivity of evidence evaluation for detecting useful innovations in resuscitation.