Large Language Models and Text Embeddings for Detecting Depression and Suicide in Patient Narratives.

Area of Science:

• Artificial Intelligence in Mental Health
• Natural Language Processing for Clinical Assessment
• Computational Psychiatry

Background:

• Large language models (LLMs) and text-embedding models offer novel approaches for analyzing narrative data in mental health.
• Previous research indicates potential for AI in assessing psychiatric patient risks.

Purpose of the Study:

• To evaluate the efficacy of LLMs and text-embedding models in detecting depression and suicide risk using Sentence Completion Test (SCT) narratives.
• To compare the performance of different LLMs and text-embedding models in identifying mental health risks.

Main Methods:

• A cross-sectional study analyzed SCT data from 1064 psychiatric patients (aged 18-39).
• LLMs (GPT-4o, Gemini-1.0-pro, GPT-3.5-turbo-16k) and text-embedding models (text-embedding-3-large, -small, ada-002) processed SCT narratives.
• Performance was measured using AUROC, balanced accuracy, and macro F1-score, focusing on self-concept, family, gender perception, and interpersonal relations narratives.

Main Results:

• LLM1 demonstrated strong zero-shot performance for depression (AUROC 0.720) and suicide risk (AUROC 0.731) using self-concept narratives.
• Few-shot learning improved LLM performance; text-embedding-3-large with extreme gradient boosting achieved the highest AUROC for depression (0.841).
• Self-concept narratives yielded the most accurate risk detections across all evaluated models.

Conclusions:

• LLMs and text-embedding models show potential for detecting depression and suicide risk in psychiatric patients via SCT narratives, especially self-concept data.
• While promising, further advancements in model performance and safety are necessary for safe clinical implementation.