A Natural Language Processing Pipeline based on the Columbia-Suicide Severity Rating Scale

Area of Science:

• Medical Informatics
• Clinical Natural Language Processing
• Public Health Surveillance

Background:

• Electronic Health Records (EHRs) have limitations in capturing patient suicidality and its severity.
• Existing diagnostic codes (ICD-9/10) are insufficient for detailed assessment of suicide risk.

Purpose of the Study:

• To develop and validate a portable Natural Language Processing (NLP) algorithm for detecting suicidal ideation (SI) and suicide-related behavior/attempts (SB/SA) within EHR data.
• To compare the performance of the NLP algorithm against traditional International Statistical Classification of Diseases (ICD) diagnostic codes.

Main Methods:

• An NLP algorithm was designed based on the Columbia-Suicide Severity Rating Scale (C-SSRS) criteria.
• Clinical notes from three major academic medical centers were manually annotated to create a "Gold Standard" for algorithm evaluation.
• The algorithm's accuracy was assessed using F1 scores, and its detection rates were compared to ICD codes across demographic groups.

Main Results:

• The NLP algorithm achieved high accuracy scores (F1: 0.86-0.97) across all participating sites.
• NLP demonstrated significantly higher detection rates for SB/SA (almost 30x) and SI (almost 10x) compared to ICD codes.
• The algorithm showed no performance bias related to race/ethnicity and performed comparably in both psychiatric and non-psychiatric EHRs.

Conclusions:

• The developed NLP algorithm accurately identifies and differentiates SI and SB/SA from clinical notes.
• NLP significantly enhances the ascertainment of suicide risk compared to current ICD coding practices.
• This algorithm offers a more effective tool for identifying at-risk individuals within EHR systems, potentially rendering ICD codes obsolete for this purpose.