The perils of multiple statistical tests: controlling for false positives

Threshold statistical testing produces errors: false positives (type I errors) and false negatives (type II errors). False positives can be quantified as the familywise error rate (FWER) or the false discovery rate (FDR). At the usual significance threshold of 0.05, the chance of a false positive is 5%. However, across a family of independent tests, the chance of at least one false positive (the FWER) is much higher, rising to >60% for 20 tests. A recent simulation study by Chen and Dexter compared different methods of controlling for false positives, which highlights the perils of multiple statistical tests. Here, we discuss their findings and examine the pros and cons of different approaches to multiple testing.