Consensus evidence evaluation in resuscitation research: analysis of Type I and Type II errors

Area of Science:

• Cardiopulmonary resuscitation (CPR) research
• Biostatistics
• Evidence-based medicine

Background:

• Consensus guidelines are crucial for implementing effective cardiopulmonary resuscitation (CPR) techniques.
• Evaluating the evidence from multiple clinical trials is complex and can be prone to errors.
• There is a need to ensure that genuine CPR advancements are identified and adopted.

Purpose of the Study:

• To statistically assess whether significant improvements in cardiopulmonary resuscitation (CPR) success rates would be recognized and incorporated into consensus guidelines.
• To determine the likelihood of false-negative evaluations (missing real CPR improvements) versus false-positive evaluations (adopting ineffective techniques).

Main Methods:

• Calculated statistical power for hypothetical clinical trials comparing experimental and control CPR, assuming a doubling of survival probabilities.
• Determined binomial distributions for statistically significant studies in series of equally powered trials.
• Compared these distributions with various expert consensus criteria for approving new CPR methods.

Main Results:

• False-negative evaluations, where true CPR survival improvements were missed, were common across realistic assumptions and consensus criteria, particularly with long-term survival data.
• False-positive evaluations were rare when at least two studies showed a statistically significant benefit.

Conclusions:

• Current evidence evaluation methods for CPR guidelines are suboptimal and can lead to missed innovations.
• A 'two and one quarter test' (at least two significant studies, comprising at least 25% of the total) is proposed to reduce Type I and Type II errors.
• Increased use of cumulative meta-analysis is recommended to improve the sensitivity of evidence evaluation for detecting CPR advancements.