Evaluation of Alignment Between Large Language Models and Expert Clinicians in Suicide Risk Assessment.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Mental Health Technology

Background:

• Large language models (LLMs) power popular chatbots such as ChatGPT, Claude, and Gemini.
• The safety and efficacy of LLM-based chatbots in responding to sensitive queries, particularly those related to suicide, require thorough evaluation.
• Understanding how these AI tools handle varying levels of suicide risk is crucial for responsible deployment.

Purpose of the Study:

• To assess whether ChatGPT, Claude, and Gemini provide direct responses to suicide-related queries.
• To determine if chatbot responses align with clinician-determined suicide risk levels.
• To compare the response patterns of different LLM-based chatbots to suicide risk queries.

Main Methods:

• Thirteen clinical experts categorized 30 hypothetical suicide-related queries into five risk levels (very high to very low).
• Each of the three LLM-based chatbots generated 100 responses per query (N=9,000 total responses).
• Responses were classified as direct (answering) or indirect (declining/referring), and analyzed using mixed-effects logistic regression.

Main Results:

• All chatbots avoided direct responses to very-high-risk queries and responded directly to all very-low-risk queries.
• Chatbots showed inconsistency in addressing intermediate suicide risk levels (low, medium, high).
• Claude was more likely than ChatGPT to provide direct responses, while Gemini was less likely.

Conclusions:

• LLM-based chatbots align with expert judgment on extreme suicide risk (very low and very high) queries.
• Inconsistent handling of intermediate suicide risk queries highlights the need for LLM refinement.
• Further development is necessary to ensure LLM-based chatbots provide safe and appropriate responses across the spectrum of suicide risk.