Illusory generalizability of clinical prediction models

Area of Science:

• Psychiatry
• Medical Informatics
• Biostatistics

Background:

• Statistical models are hoped to enhance medical treatment decision-making.
• Model development often relies on limited datasets or clinical contexts due to data costs and scarcity.
• This reliance raises questions about the generalizability of predictive models in healthcare.

Purpose of the Study:

• To scrutinize the optimism surrounding statistical models in medical decision-making.
• To examine the performance of a machine learning model across independent clinical trials for schizophrenia.
• To assess the generalizability of predictive models for antipsychotic medication outcomes.

Main Methods:

• A machine learning model was developed using data from a clinical trial for schizophrenia.
• The model's predictive accuracy was tested on out-of-sample data from independent clinical trials.
• Data pooling across trials was explored to improve out-of-sample prediction performance.

Main Results:

• The machine learning model achieved high accuracy within the dataset it was developed on.
• The model performed no better than chance when applied to independent, out-of-sample clinical trials.
• Pooling data across trials did not enhance the model's predictive performance on unseen data.

Conclusions:

• Machine learning models for predicting schizophrenia treatment outcomes are highly context-dependent.
• These models demonstrate limited generalizability across different clinical trials or patient populations.
• The findings challenge the broad applicability of current predictive models in personalized psychiatric treatment.