Controlling the familywise error rate in functional neuroimaging: a comparative review

Functional neuroimaging data embodies a massive multiple testing problem, where 100,000 correlated test statistics must be assessed. The familywise error rate, the chance of any false positives is the standard measure of Type I errors in multiple testing. In this paper we review and evaluate three approaches to thresholding images of test statistics: Bonferroni, random field and the permutation test. Owing to recent developments, improved Bonferroni procedures, such as Hochberg's methods, are now applicable to dependent data. Continuous random field methods use the smoothness of the image to adapt to the severity of the multiple testing problem. Also, increased computing power has made both permutation and bootstrap methods applicable to functional neuroimaging. We evaluate these approaches on t images using simulations and a collection of real datasets. We find that Bonferroni-related tests offer little improvement over Bonferroni, while the permutation method offers substantial improvement over the random field method for low smoothness and low degrees of freedom. We also show the limitations of trying to find an equivalent number of independent tests for an image of correlated test statistics.