Related Experiment Video
Updated: Oct 22, 2026
Implementation of a Real-Time Psychosis Risk Detection and Alerting System Based on Electronic Health Records using CogStack
Published on: May 15, 2020
Development and internal validation of machine learning models for identifying suicidal behavior in inpatients with
Sixiang Liang1, Qitong Jiang1, Jinhe Zhang1
1The National Clinical Research Center for Mental Disorders & Beijing Key Laboratory of Mental Disorders, Beijing Anding Hospital, Capital Medical University & the Advanced Innovation Center for Human Brain Protection, Capital Medical University, Beijing, China.
Background:
Suicidal behavior (SB) is a major concern in mood disorders. Models developed from retrospective records are vulnerable to ambiguous targets, data leakage, and inappropriate handling of class imbalance. We developed and internally validated a model to identify the risk of suicidal behavior in patients with mood disorders upon admission.
Methods:
The dataset included 1, 099 inpatients with ICD-10 mood disorders admitted to Beijing Anding Hospital in 2022. Exclusion of 20 records with more than 20% row-level missingness and one record without an outcome left 1, 078 patients, including 126 with SB. Current SB comprised documented suicidal ideation, planning, or attempt within the three months before index admission, including the admission assessment. Previous SB and previous suicide attempt (SA) referred to events more than three months before admission. Elastic Net selected 14 predictors from 65 candidates in the development set. We compared 13 classifiers from six model families. All preprocessing and resampling were confined to training folds, and evaluation used only observed patients. Assessment included repeated fivefold cross-validation, probability calibration, bootstrap confidence intervals, precision-recall and decision-curve analyses, clinical baselines, and sensitivity analyses.
Results:
The analytic cohort comprised 952 SB-negative and 126 SB-positive patients (11.69%). The development set contained 864 patients (97 SB-positive), and the observed test set contained 214 patients (29 SB-positive). Calibrated linear discriminant analysis (LDA) provided the most consistent balance of discrimination and calibration. In repeated cross-validation, mean AUC was 0.721 ± 0.052, mean AUPRC was 0.268 ± 0.060, and mean Brier score was 0.095. In the observed test set, AUC was 0.704 (0.597-0.803), AUPRC was 0.283 (0.198-0.453), and Brier score was 0.111. At the training-derived threshold of 0.111, sensitivity was 0.552 and specificity was 0.659. The leading SHAP contributions were onset-polarity category, previous SA, age, and previous SB, but all attributions were associational.
Conclusions:
Model performance was acceptable. These internally validated findings are exploratory and require prospective assessment, standardized outcome measurement, and external validation before clinical use.