Cluster-based permutation tests of MEG/EEG data do not establish significance of effect latency or location

Cluster-based permutation tests are gaining an almost universal acceptance as inferential procedures in cognitive neuroscience. They elegantly handle the multiple comparisons problem in high-dimensional magnetoencephalographic and EEG data. Unfortunately, the power of this procedure comes hand in hand with the allure for unwarranted interpretations of the inferential output, the most prominent of which is the overestimation of the temporal, spatial, and frequency precision of statistical claims. This leads researchers to statements about the onset or offset of a certain effect that is not supported by the permutation test. In this article, we outline problems and common pitfalls of using and interpreting cluster-based permutation tests. We illustrate these with simulated data in order to promote a more intuitive understanding of the method. We hope that raising awareness about these issues will be beneficial to common scientific practices, while at the same time increasing the popularity of cluster-based permutation procedures.