External validation improves generalizability, replicability and reproducibility in predictive models for...

Poor generalizability continues to hinder mapping brain activity to behavior using human neuroimaging data. A potential solution is predictive modeling, which evaluates generalizability in unseen data from the same dataset. However, predictive models often fail external validation-a stricter test of generalizability involving evaluation in an independent dataset. In this Perspective, we explain how evaluating generalizability via external validation can improve replicability (illusory generalizability and bias) and reproducibility (data leakage and data manipulations). We also provide advice on statistical power, dataset shift and training models. A model's success in external validation provides evidence for its generalizability, but interpretations depend on the population characteristics of the training and external datasets. Even failed external validation constitutes an opportunity for scientific insight or methodological adjustments. Sharing data and models, combined with standalone external validation studies, will increase the prevalence of external validation. In turn, an increased focus on external validation can drive more generalizable, replicable and reproducible neuroimaging results.