Bias in a prehospital esophageal detector device trial: lessons learned

Our goals were to determine whether selection bias occurred in a prehospital study comparing an esophageal detector device (EDD) to a disposable capnometer for detecting esophageal intubation, and to determine whether such a bias would have changed the study's conclusions about EDD effectiveness.

In a study of patients requiring prehospital intubation, we determined the sensitivity, specificity and predictive values of the EDD for detecting esophageal intubation. We then compared intubation success rate in patients who were enrolled in the study (n = 129) to that in eligible patients who were excluded from it (n = 107). After finding that the incidence of failed intubation was higher in the "excluded" group, we used sensitivity and specificity parameters derived from the study population to assess whether EDD test characteristics would differ in studied vs. excluded patients.

The first intubation attempt was successful in 125 of 129 study patients and 76 of 107 excluded patients (97% vs. 71%, p = 0.03), confirming the presence of selection bias. The negative predictive value of the EDD for esophageal intubation was 98% in the study cohort and would have been 77% in patients like those excluded (i.e., difficult intubation cases).

The high "first attempt" intubation success rate seen in this study was due to selective exclusion of failed intubations. This selection bias led to a clinically important overestimation of the EDD's negative predictive value. Bias may substantially alter the estimations of test accuracy reported in scientific studies. To reduce the chance of unrecognized selection bias in studies of diagnostic tests, investigators must determine whether recruited subjects resemble patients in whom the test will ultimately be used.