Registration and multiple outcome testing in the HEALing communities study

The HEALing Communities Study (HCS) is a randomized control trial that began with one prespecified, registered opioid overdose outcome. Just before the study began, three additional overdose outcomes were described in a public protocol. After the study had commenced data collection, two documents were published that described more than 20 additional sub-measures. Following the reporting of a null result for the registered primary outcome, investigators from the HCS team delved into these documents and published papers reporting isolated statistically significant results. Statistical tests of these 20-plus outcomes should be considered a family of tests pertaining to the effects of the HCS intervention on drug overdose events and deaths and the family-wise error rate corrected when presenting P-values. Like most trials published in academic journals, the HCS researchers have not corrected for multiplicity in their data analyses of these outcomes and the conclusions drawn about the effectiveness of the intervention are likely based on results that are subject to Type I error. The best point in the academic production process at which to correct these problems is journal peer review. Journal editors could limit the type of causal inferences investigators can draw from exploratory analyses in the discussion section of a paper, require them to conduct corrections for multiplicity and/or clearly distinguish any uncorrected P-values from corrected ones, and only permit reporting of effect estimates and confidence intervals for secondary outcomes.