Replicability and generalizability in population psychiatric neuroimaging.

Area of Science:

• Neuroscience
• Psychiatry
• Data Science

Background:

• Cross-sectional studies on mental health and brain function often use small samples, limiting statistical power and replicability.
• Small effect sizes in brain-wide associations necessitate large sample sizes (thousands) for reliable results.
• Historically, underpowered samples have hampered progress in understanding brain-behavior relationships in mental health.

Purpose of the Study:

• To highlight how small sample sizes impede replicability in mental health-brain function studies.
• To provide sample size targets for achieving statistical power based on effect size benchmarks.
• To discuss generalizability challenges and the risk of shortcut learning in psychiatric neuroimaging.

Main Methods:

• Analysis of historical association studies in mental health and brain function.
• Benchmarking of sample size requirements for statistical power and replicability.
• Conceptual analysis of generalizability issues and shortcut learning in machine learning models for psychiatric neuroimaging.

Main Results:

• Small sample sizes are a primary driver of poor replicability in mental health neuroimaging.
• Larger samples improve replicability but do not guarantee generalizability to target populations.
• Shortcut learning can lead to spurious associations between brain data and mental health phenotypes.

Conclusions:

• Future epidemiological studies in mental health require large, diverse, and comprehensively assessed samples.
• Addressing generalizability failures, such as shortcut learning, is critical for valid brain-based predictions.
• Robust findings in psychiatric neuroimaging depend on overcoming limitations of small, underpowered studies.