The Benjamini-Hochberg procedure can fail to control the FDR for correlated two-sided Gaussian tests

The Benjamini-Hochberg (BH) procedure is the standard method for controlling the false discovery rate (FDR) in large-scale studies involving testing multiple hypotheses. Whether the BH procedure controls the FDR for every correlation structure among two-sided Gaussian tests has been a long-standing question, and a positive answer has been believed. Here, we settle the question negatively, by providing examples of Gaussian factor models for which the BH method fails to control the FDR at the nominal level. We prove mathematically that the violation holds, and also provide simulation results which are consistent with the theory. The level of violation can be characterized as mild but nonnegligible. In one of our examples, empirical FDR levels of ≈0.0106 and ≈0.053 are achieved for nominal FDR levels 0.01 and 0.05, respectively. Larger violations are also possible. The violation appears most prominently for large numbers (thousands) of tests; though we also present an example with a moderate number of 85 tests. We also briefly discuss the implications of our findings.