When Is Enough, Enough? When Are More Observational Epidemiologic Studies Needed to Resolve a Research Question:...

Research reproducibility is vital for translation of epidemiologic findings. However, repeated studies of the same question may be undertaken without enhancing existing knowledge. To identify settings in which additional research is or is not warranted, we adapted research synthesis metrics to determine number of additional observational studies needed to change the inference from an existing meta-analysis.

The fail-safe number (FSN) estimates number of additional studies of average weight and null effect needed to drive a statistically significant meta-analysis to null (P ≥ 0.05). We used conditional power to determine number of additional studies of average weight and equivalent heterogeneity to achieve 80% power in an updated meta-analysis to detect the observed summary estimate as statistically significant. We applied these metrics to a curated set of 98 meta-analyses on biomarkers and cancer risk.

Both metrics were influenced by number of studies, heterogeneity, and summary estimate size in the existing meta-analysis. For the meta-analysis on Helicobacter pylori and gastric cancer with 15 studies [OR = 2.29; 95% confidence interval (CI), 1.71-3.05], FSN was 805 studies, supporting futility of further study. For the meta-analysis on dehydroepiandrosterone sulfate and prostate cancer with 7 studies (OR = 1.29; 95% CI, 0.99-1.69), 5 more studies would be needed for 80% power, suggesting further study could change inferences.

Along with traditional assessments, these metrics could be used by stakeholders to decide whether additional studies addressing the same question are needed.

Systematic application of these metrics could lead to more judicious use of resources and acceleration from discovery to population-health impact.