Machine Learning-Based Evaluation of Suicide Risk Assessment in Crisis Counseling Calls.

Area of Science:

• Artificial Intelligence
• Mental Health Technology
• Clinical Psychology

Background:

• Suicide risk assessment is crucial in crisis counseling, mandated by standards.
• Current quality improvement relies on manual, slow, and unscalable human evaluation of conversations.
• Limitations in human evaluation hinder efforts to increase the frequency of risk assessments.

Purpose of the Study:

• To develop and evaluate machine learning (ML) models for automatic detection of suicide risk assessment in crisis counseling.
• To provide a scalable tool for quality improvement in crisis intervention.

Main Methods:

• A dataset of 476 crisis counseling calls (193,257 statements) was manually coded for risk assessment elements.
• A transformer-based ML model was fine-tuned using this labeled data.
• The model was evaluated using separate training, validation, and test datasets.

Main Results:

• The ML model demonstrated high consistency with human raters, achieving 98% agreement with human interrater reliability for detecting any risk assessment.
• The model achieved an average F1 score of 0.86 at the call level and 0.66 at the statement level.
• Performance varied for specific labels due to low base rates.

Conclusions:

• Machine learning models can reliably identify suicide risk assessment in crisis counseling conversations.
• This technology presents a significant opportunity to scale quality improvement initiatives in mental health crisis services.