Navigating extreme class imbalance in suicide risk prediction

Area of Science:

• Public Health
• Data Science
• Clinical Informatics

Background:

• Suicide risk models face implementation challenges due to discrepancies between development and real-world conditions.
• Arbitrary classification thresholds limit the interpretability of predictive models and their performance statistics.
• Understanding how class imbalance affects predictive model performance is crucial for accurate suicide risk assessment.

Purpose of the Study:

• To explore the impact of varying class imbalance ratios on the performance of regression-based suicide risk prediction models.
• To investigate how training sample composition, time horizons, and patient characteristics influence model performance.
• To enhance the interpretability and reliability of suicide risk prediction models.

Main Methods:

• Utilized a large dataset of 1,649,577 patients from the Maryland Suicide Data Warehouse (MSDW) spanning 2016-2020.
• Employed a cross-validated framework to assess model performance, stratifying by data sources (MHCC, HSCRC).
• Analyzed the association between class imbalance, time horizons, patient characteristics (age, care utilization, social needs) and model performance metrics (AUROC, AUPRC).

Main Results:

• Area Under the Receiver Operating Characteristic Curve (AUROC) did not consistently vary with training sample imbalance or time horizon.
• Area Under the Precision-Recall Curve (AUPRC) showed a direct association with sample imbalance and time horizon, increasing with greater imbalance.
• Stratified analyses revealed that AUPRC significantly improved for specific patient cohorts, such as those seen in emergency rooms, inpatient settings, or with documented social needs, while performance worsened for patients under 18.

Conclusions:

• Regression models for suicide risk prediction demonstrate improved AUPRC when developed with more realistic conditions, including class imbalance and less restrictive time horizons.
• Training models on specific clinical cohorts (defined by age, care utilization, social needs) can lead to significantly different precision and recall estimates, but not AUROC.
• Adjusting model development parameters based on real-world data characteristics is essential for robust and interpretable suicide risk prediction.