Predictive Accuracy of Natural Language Processing Extracted 3-Step Theory of Suicide Factor Scores Derived From...

To compare predictive accuracy of 3-step theory of suicide (3ST) factor scores derived from natural language processing of Veterans Health Administration (VHA) clinical progress notes versus a model that underlies VHA's Recovery Engagement and Coordination for Health-Veterans Enhanced Treatment (REACH VET) program retrained to predict the combined outcome of suicide attempt or suicide death, and to compare characteristics of patients accurately predicted by both approaches.

As health systems incorporate risk prediction models to guide suicide prevention efforts, it is important to evaluate their predictive accuracy and to consider the benefits of different modeling approaches.

A comparative cohort design in which both risk prediction approaches were evaluated for the same random sample (n=162,132) of VHA patients alive on May 1, 2018, who had clinical encounters during the 4 weeks before that date.

At the highest risks (top 1%-5%), the model based on REACH VET variables outperformed the 3ST approach in terms of positive predictive value and false-negative rate. Among patients who attempted or died by suicide, uniquely identified by the 3ST approach and not by the retrained REACH VET model, none had attempted suicide during the prior 6 months, emergency department visits during the prior month, discharges from mental health hospitalizations during the prior 12 months, or a diagnosis of bipolar disorder during the prior 24 months.

Additional research is recommended to further prepare 3ST factor scores based on NLP of clinical progress notes for use in clinical decision-making.