Evaluation of suicide behavior screening tools using machine learning and variable importance measures

Suicide screening in military primary care settings is often conducted with a small number of self-report questions. The PRImary care Screening Methods (PRISM) study investigates how to enhance this screening's predictive validity by supplementing the standard approach with additional questions. The present research uses machine learning, specifically random forests implementing undersampling with discretized data, to characterize the most important risk/protective factors. An analysis of 1522 PRISM participants using 875 features identified suicide attempts up to 12 months post-baseline with 68.7% out-of-bag accuracy. The top five features identified using variable importance measures (VIMs) included: (1) "trouble falling or staying asleep, or sleeping too much"; (2) "have had nightmares about it or thought about it when you did not want to" with respect to a traumatic event; (3) "organization of time" as a life stressor; (4) "have you ever had thoughts of killing yourself"; and (5) "feel like people laugh at you." For each of these, bootstrapped risk ratios revealed that endorsement of the first response option (indicating strong disagreement) served as a protective factor. The most important risk factor identified-endorsing "feel like people laugh at you" with the last response option (indicating strong agreement)-increased both the sensitivity and specificity of primary care suicide screening when used as a supplement to the standard approach. These findings highlight the under-appreciated role of self-conscious emotions and sleep dysregulation in suicidal behavior and reveal the utility of machine learning methods for improving the identification of those at risk for suicidal behavior.