A citation analysis of (f)MRI papers that cited Lieberman and Cunningham (2009) to justify their statistical...

In current neuroimaging studies, the mainstream practice is to report results corrected for multiple comparisons to control for false positives. In 2009, Lieberman and Cunningham published a highly cited report that promotes the use of uncorrected statistical thresholds to balance Types I and II error rates. This paper aims to review recent studies that cited this report, investigating whether the citations were to justify the use of uncorrected statistical thresholds, and if their uncorrected thresholds adhered to the recommended defaults.

The Web of Science Core Collection online database was queried to identify original articles published during 2019-2022 that cited the report.

It was found that the majority of the citing papers (152/225, 67.6%) used the citation to justify their statistical threshold setting. However, only 19.7% of these 152 papers strictly followed the recommended uncorrected P (Punc) < 0.005, k = 10 (15/152, 9.9%) or Punc < 0.005, k = 20 (15/152, 9.9%). Over half (78/152, 51.3%) used various cluster-extent based thresholds with Punc, with the predominant choices being Punc < 0.001, k = 50 and Punc < 0.001, k = 10, mostly without justifying their deviation from the default. Few papers matched the voxel size and smoothing kernel size used by the simulations from the report to derive the recommended thresholds.

This survey reveals a disconnect between the use and citation of Lieberman and Cunningham's report. Future studies should justify their chosen statistical thresholds based on rigorous statistical theory and study-specific parameters, rather than merely citing previous works. Furthermore, this paper encourages the neuroimaging community to publicly share their group-level statistical images and metadata to promote transparency and collaboration.