Human-in-the-Loop Suicide Risk Detection and Clinical Escalation in a Generative AI Mental Health Chatbot:...

Mental illness is the leading cause of disability worldwide, affecting more than one billion individuals, and suicide accounts for 720,000 to 800,000 deaths annually. While AI-native mental health chatbots have the potential to fill a sizable gap between treatment need and workforce capacity, their rapid proliferation has outpaced the development of scalable procedures to identify, detect, and respond to suicidal ideation. Despite AI best practice recommendations to incorporate human oversight for safety monitoring and intervention, studies of procedures for human clinical oversight and crisis response are scarce.

To implement a human-in-the-loop suicide risk screening, detection, and clinical escalation protocol in a generative AI mental health chatbot.

In an implementation study of a human-in-the-loop suicide risk detection and escalation protocol, adults were recruited online to participate in a single-arm trial of an AI-native mental health chatbot. Participants completed a pre-treatment screening using the Patient Health Questionnaire-9 (PHQ-9). In Phase 1, individuals who endorsed suicidal ideation were escalated to a human clinician for further assessment, safety planning and provision of referral resources, as appropriate. In Phase 2, escalation was triggered by either PHQ-9 responses or detection of spontaneous disclosure of implicit or explicit suicidality in conversational AI interactions. In both phases, the Ask Suicide-Screening Questionnaire (ASQ) guided human-led assessment and risk stratification. Prevalence of suicidal ideation was assessed, along with the rate of successful human outreach, time from suicidality risk detection to human outreach and results of risk stratification evaluation.

In Phase 1, 624 participants completed suicide screening, of whom 81 (13.0%) endorsed PHQ-9 item 9 (score >1). Outreach was initiated a mean of 11.2 minutes after the alert (SD=13.2; range 1-54 minutes). A clinician successfully reached 25/81 participants (30.9%). In Phase 2, 650 participants completed screening, of whom 133 (20.5%) screened positive at baseline. Across the 30 day post-baseline observation period, 284 risk-alert events were generated among 172 participants (26.5%), including 50 conversational suicidal ideation events. Clinicians successfully contacted 73/172 participants (42.4%). Time from risk detection to clinician outreach initiation was an average of 11.5 minutes (SD 11.7; range 1-58 minutes).

Proactive, universal suicide risk screening and conversational detection using a human-in-the-loop protocol can be operationally implemented for individuals utilizing a Generative AI mental health chatbot. A subgroup of those who were responsive to human clinical outreach described the approach as acceptable and helpful. Given that contact and clinical assessment were not completed for a sizable proportion of alerted participants, further research is needed to optimize assessment capabilities for those who could not be reached, and characterize the impact of these procedures on clinical outcomes. Nevertheless, the safety procedures reported herein provide a framework from which purpose-built AI-native mental health chatbots can build and refine human-in-the-loop safety responses and protocols.