Development and Internal Validation of an Interpretable Machine Learning Model for Identifying Past-Year Nonsuicidal...

Nonsuicidal self-injury (NSSI) is common among adolescents with depressive disorders and is associated with substantial clinical burden. Machine learning may assist in identifying multidimensional patterns associated with NSSI, although its incremental value beyond direct clinical assessment remains uncertain.

This study compares 7 supervised machine learning model variants for classifying past-year NSSI status among adolescents with depressive disorders and conducts detailed analyses of interpretability, sensitivity, subgroup performance, and clinical utility using a random forest model as the focal exploratory model.

We retrospectively identified 437 adolescents aged 10-19 years who received inpatient or outpatient psychiatric care at 4 hospitals in Zhejiang Province, China, between January 2021 and December 2023. A total of 410 eligible participants were included, of whom 274 reported NSSI during the preceding year. The primary analysis included 67 assessment-time sociodemographic, physiological, biochemical, and clinical-behavioral features. The sample was divided into a training set (287/410, 70%) and an untouched held-out test set (123/410, 30%) using stratified random sampling. Hyperparameters were optimized using 5-fold cross-validation within the training set. Performance was evaluated using discrimination, classification metrics, calibration, and decision-curve analysis. Random forest model behavior was examined using Gini importance, permutation importance, and Shapley Additive Explanations (SHAP).

The radial basis function support vector machine achieved the highest mean cross-validated area under the receiver operating characteristic curve (AUROC) in the training set (0.720), whereas the random forest showed the highest observed AUROC and area under the precision-recall curve (AUPRC) and the lowest Brier score in the held-out test set. The random forest achieved an AUROC of 0.681 (95% CI 0.583-0.778), an AUPRC of 0.808 (95% CI 0.741-0.874), an accuracy of 0.667 (95% CI 0.585-0.748), a sensitivity of 0.720 (95% CI 0.622-0.817), a specificity of 0.561 (95% CI 0.415-0.707), and a Brier score of 0.212 (95% CI 0.188-0.236). Its calibration slope was 1.013. Pairwise AUROC differences between the random forest and alternative models were not statistically significant after Holm correction (Holm-adjusted P values ranged from .17 to .81). The full random forest model did not outperform a suicidal-ideation-only model (AUROC 0.701), whereas excluding suicidal ideation and previous suicide attempt reduced the AUROC to 0.540. A biochemical-only model showed near-chance discrimination (AUROC 0.513). Decision-curve analysis indicated that the random forest provided greater net benefit than the assess-all and assess-none strategies across threshold probabilities of approximately 0.51-0.80. Suicidal ideation showed the most consistent importance across Gini, SHAP, and permutation analyses.

The random forest had the highest observed AUROC and AUPRC and the lowest Brier score in the held-out test set among the evaluated algorithms, although the pairwise AUROC differences were not statistically significant after correction for multiple comparisons. Suicidality-related information contributed substantially to model discrimination, whereas routinely collected biochemical variables did not demonstrate clear incremental value. Because this was a retrospective cross-sectional classification study, the findings do not establish prospective risk prediction. External and prospective temporal validation is required before clinical implementation.