Generalizability of Risk Models for Treatment-Resistant Depression Across Three Health Systems.

Area of Science:

• Clinical Informatics
• Psychiatry
• Machine Learning in Healthcare

Background:

• Effective management of treatment-resistant major depressive disorder (TRD) necessitates early identification of at-risk individuals.
• Current strategies for TRD management are evolving, underscoring the need for predictive tools.
• Early identification of patients likely to develop TRD is essential for timely intervention.

Purpose of the Study:

• To develop and validate machine learning models for predicting treatment-resistant depression using electronic health records (EHR).
• To assess the generalizability and performance of predictive models across diverse healthcare systems.
• To identify key features influencing model performance and discordance.

Main Methods:

• Extracted EHR data from three distinct health systems (MGB, VUMC, GC) for patients with major depressive disorder (MDD).
• Defined treatment-resistant depression (TRD) based on specific treatment criteria (e.g., ECT, ketamine, multiple failed trials).
• Applied L1-regularized regression and random forest analyses to sociodemographic, medication, and diagnostic code data.

Main Results:

• Model discrimination performance varied, with Area Under the ROC curves (AUROCs) ranging from 0.51-0.58 in external validation.
• Concordance Correlation Coefficients (CCCs) between models were low (0.13-0.38), indicating limited agreement across health systems.
• Age and sex were primary predictors of model discordance, suggesting potential biases.

Conclusions:

• Linear models showed consistent aggregate but discordant individual performance across health systems.
• Improving predictive accuracy may require incorporating data types beyond standard EHR information.
• Further research is needed to ensure equitable performance across subgroups and to evaluate clinical utility for early identification.