Sampling inequalities affect generalization of neuroimaging-based diagnostic classifiers in psychiatry

Area of Science:

• Psychiatry
• Neuroimaging
• Machine Learning
• Data Science

Background:

• Machine learning (ML) models show promise for psychiatric disorder diagnosis.
• Poor generalizability of ML models hinders their clinical application.

Purpose of the Study:

• To quantitatively assess global and regional sampling issues in neuroimaging-based ML models for psychiatric diagnosis.
• To develop a rating system for evaluating the quality of these ML models.

Main Methods:

• Meta-research assessment of 476 psychiatric studies (n=118,137) using neuroimaging data.
• Quantitative analysis of sampling inequality using the Gini coefficient.
• Development of a 5-star rating system for ML model quality.

Main Results:

• Significant global sampling inequality (G=0.81) was observed, correlated with national economic levels.
• Higher sampling inequality was associated with higher reported classification accuracy.
• Prevalence of issues like lack of independent testing (84.24%), improper cross-validation (51.68%), and poor transparency/availability (87.8%/80.88%).
• Model performance decreased with independent cross-country validation.

Conclusions:

• Improving economic equality in sampling is vital for enhancing the generalizability and clinical translation of ML diagnostic models.
• Addressing sampling biases and methodological limitations is crucial for reliable psychiatric diagnosis using ML.