The problem of multiple inference in studies designed to generate hypotheses

Area of Science:

• Epidemiology
• Biostatistics

Background:

• Epidemiologic studies often assess numerous risk factors and disease outcomes simultaneously.
• Multiple univariate hypothesis testing is inadequate for inference in such complex scenarios, leading to potential false positives.

Purpose of the Study:

• To evaluate methods for managing and interpreting associations in large-scale epidemiologic studies.
• To propose a robust approach for hypothesis generation when exploring multiple associations.

Main Methods:

• Comparison of observed p-value distributions to theoretical or empirical distributions.
• Consideration of statistical criteria like Bonferroni adjustment, sample splitting, Bayesian inference, and decision theory.
• Application of empirical Bayes techniques for ranking associations by investigative priority.

Main Results:

• Existing methods for estimating true positives are insufficient to distinguish true from false associations.
• The proposed method involves reporting all associations and prioritizing them using empirical Bayes.

Conclusions:

• Reporting all associations, regardless of significance, is recommended.
• Empirical Bayes techniques offer a valuable approach for prioritizing associations in hypothesis-generating epidemiologic research.