Complex modeling with detailed temporal predictors does not improve health records-based suicide risk prediction

Area of Science:

• Computational psychiatry
• Clinical informatics
• Machine learning in healthcare

Background:

• Suicide risk prediction models are crucial for targeted interventions.
• Complex machine learning models are often presumed superior to simpler models due to transparency and explainability.
• The performance of complex models versus simpler models in predicting suicidal behavior requires empirical validation.

Purpose of the Study:

• To compare the performance of complex machine learning models against simpler logistic regression models for predicting suicidal behavior.
• To evaluate whether complex models with numerous predictors offer significant advantages over simpler models in clinical utility.
• To assess model performance across diverse demographic subgroups.

Main Methods:

• Trained and evaluated suicide risk prediction models using data from 25,800,888 mental health visits across 7 health systems.
• Compared random forest, artificial neural network, and ensemble models against logistic regression models.
• Utilized 1500 temporally defined predictors for complex models and a subset for logistic regression.

Main Results:

• All evaluated models demonstrated strong performance, with area under the receiver operating curve (AUC) ranging from 0.794 to 0.858.
• Ensemble models showed the best performance, but improvements over a logistic regression model with 100 predictors were marginal (0.006–0.020 AUC increase).
• Model performance was consistent across various metrics and demographic subgroups, including race, ethnicity, and sex.

Conclusions:

• Simpler parametric models, such as logistic regression, perform comparably to more complex machine learning methods for suicide risk prediction.
• The minimal performance gains of complex models suggest that simpler, more interpretable models are viable for clinical implementation.
• Easier implementation of simpler models can facilitate their integration into routine clinical practice for identifying individuals at risk of suicide.