Detection of Youth Suicide Interventions in Clinical Record Text using an Open-Source Language Model

Area of Science:

• Medical Informatics
• Public Health
• Artificial Intelligence in Healthcare

Background:

• Youth suicide is a critical public health issue.
• Emergency departments are key points for identifying and intervening in youth suicidality.
• Documenting suicide prevention interventions in clinical notes is essential for care quality.

Purpose of the Study:

• To develop and validate an automated method for detecting youth suicide prevention interventions in emergency department (ED) notes.
• To assess the performance of an open-source language model in identifying specific interventions.
• To identify factors associated with omitted interventions in ED encounters for youth suicidality.

Main Methods:

• Utilized expert review to classify four key interventions (Lethal Means Restriction, Hotline, Outpatient Referral, Safety Planning) in 1,794 ED notes.
• Employed the open-source language model Llama3.3-70B to generate Likert scores for intervention presence, achieving high AUROC values for each intervention.
• Analyzed 6,687 notes from 723 encounters to determine predictors of intervention omission.

Main Results:

• The language model demonstrated high accuracy in detecting interventions, with AUROC values ranging from 0.935 to 0.980.
• Youth with prior ED visits, missing screening questions, or presenting with ideation (compared to acts/attempts) had increased odds of omitted interventions.
• Findings suggest systematic clinical text analysis can reveal targets for improving suicide prevention care.

Conclusions:

• Automated analysis of clinical text using open-source language models is feasible and accurate for identifying youth suicide prevention interventions.
• The study identified specific patient characteristics and encounter types associated with gaps in documented interventions.
• This approach can inform decision support tools and enhance evidence-based suicide prevention strategies for young people.